Jul 28 00:46:08.332253 ip-10-0-131-111 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 00:46:08.727808 ip-10-0-131-111 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 00:46:08.727808 ip-10-0-131-111 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 00:46:08.727808 ip-10-0-131-111 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 00:46:08.727808 ip-10-0-131-111 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 00:46:08.727808 ip-10-0-131-111 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 00:46:08.731393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.731303    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 00:46:08.741112 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741087    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:08.741112 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741107    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:08.741112 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741112    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:08.741112 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741116    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:08.741112 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741120    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741125    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741129    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741133    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741137    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741141    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741145    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741151    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741154    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741158    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741162    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741166    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741170    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741174    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741178    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741181    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741185    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741198    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741203    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741206    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:08.741414 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741210    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741214    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741218    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741222    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741226    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741229    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741233    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741237    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741242    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741245    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741250    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741254    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741258    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741263    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741268    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741273    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741278    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741282    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741286    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741290    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:08.742239 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741294    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741299    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741303    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741308    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741312    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741317    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741321    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741325    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741329    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741334    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741338    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741342    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741347    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741351    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741355    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741360    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741364    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741368    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741372    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741376    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:08.743098 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741380    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741384    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741388    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741394    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741401    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741408    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741413    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741418    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741422    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741427    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741431    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741435    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741439    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741444    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741449    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741454    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741459    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741463    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741467    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:08.743925 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741473    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741477    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741481    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741485    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741489    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741494    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741498    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741502    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741506    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741510    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741516    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741521    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741525    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741531    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741538    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.741545    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742724    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742735    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:08.744761 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742741    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742749    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742755    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742760    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742766    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742770    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742775    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742780    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742785    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742789    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742794    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742799    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742803    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742808    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742812    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742817    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742821    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742825    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742829    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:08.745556 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742834    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742838    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742843    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742847    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742851    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742855    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742859    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742863    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742867    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742871    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742876    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742880    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742885    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742890    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742895    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742899    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742903    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742907    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742912    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742916    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:08.746212 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742921    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742925    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742929    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742952    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742957    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742961    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742965    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742969    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742973    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742978    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742982    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742986    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742990    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742994    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.742999    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743003    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743008    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743015    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743020    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743024    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:08.746697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743028    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743033    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743038    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743043    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743048    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743052    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743057    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743062    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743067    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743072    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743076    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743081    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743085    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743089    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743093    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743097    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743103    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743108    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743111    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743115    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:08.747293 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743119    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743123    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743129    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743133    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743137    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743141    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743146    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743150    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743154    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743160    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743164    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743168    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743173    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743177    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743185    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743191    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743197    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743202    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743862    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743871    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:08.747824 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743876    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743880    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743884    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743889    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743893    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743897    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743901    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743906    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743910    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743914    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743920    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743924    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743928    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743947    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743953    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743957    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743962    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743966    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743970    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:08.748540 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743974    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743978    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743982    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743987    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743991    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.743995    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744000    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744007    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744013    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744019    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744025    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744029    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744034    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744038    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744043    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744048    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744052    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744056    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744061    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:08.749035 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744065    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744069    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744073    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744077    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744081    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744086    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744090    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744095    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744099    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744104    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744109    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744113    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744118    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744122    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744126    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744130    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744135    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744139    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744143    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744147    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:08.749564 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744151    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744156    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744160    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744165    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744169    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744173    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744177    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744181    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744186    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744190    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744194    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744198    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744205    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744210    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744214    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744218    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744222    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744226    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744231    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744235    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:08.750113 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744239    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744243    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744249    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744254    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744261    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744266    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744270    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744275    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744280    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744284    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744288    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744292    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744299    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744303    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744308    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744312    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744316    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744321    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.744325    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744431    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 00:46:08.750591 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744445    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744458    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744464    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744472    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744477    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744484    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744491    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744496    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744501    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744506    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744512    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744517    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744522    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744527    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744531    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744538    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744543    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744549    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744554    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744560    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744565    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744570    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744577    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 00:46:08.751087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744582    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744590    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744596    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744601    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744606    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744610    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744615    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744620    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744627    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744632    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744637    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744641    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744646    2564 flags.go:64] FLAG: --enable-server="true"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744651    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744659    2564 flags.go:64] FLAG: --event-burst="100"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744664    2564 flags.go:64] FLAG: --event-qps="50"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744669    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744674    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744678    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744685    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744690    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744694    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744700    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744705    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744709    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 00:46:08.751761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744715    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744719    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744726    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744731    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744736    2564 flags.go:64] FLAG: --feature-gates=""
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744742    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744747    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744752    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744758    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744764    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744769    2564 flags.go:64] FLAG: --help="false"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744774    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-111.ec2.internal"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744779    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744784    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744789    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744795    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744800    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744805    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744810    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744815    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744819    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744824    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744830    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744835    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 28 00:46:08.752452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744840    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744845    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744850    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744854    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744858    2564 flags.go:64] FLAG: --lock-file=""
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744863    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744868    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744873    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744882    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744888    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744894    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744899    2564 flags.go:64] FLAG: --logging-format="text"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744904    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744909    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744913    2564 flags.go:64] FLAG: --manifest-url=""
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744918    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744926    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744946    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744953    2564 flags.go:64] FLAG: --max-pods="110"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744958    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744963    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744968    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744972    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744977    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744981    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 00:46:08.753109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744986    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.744997    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745002    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745007    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745012    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745016    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745025    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745030    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745034    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745039    2564 flags.go:64] FLAG: --port="10250"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745044    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745049    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a3f4e5a968acb162"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745054    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745059    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745063    2564 flags.go:64] FLAG: --register-node="true"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745069    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745075    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745085    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745090    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745095    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745100    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745105    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745110    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745115    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745121    2564 flags.go:64] FLAG: --runonce="false"
Jul 28 00:46:08.753713 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745127    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745132    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745137    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745142    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745147    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745151    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745156    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745161    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745166    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745171    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745176    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745180    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745185    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745190    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745194    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745203    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745208    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745212    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745219    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745223    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745228    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745232    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745237    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745242    2564 flags.go:64] FLAG: --v="2"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745250    2564 flags.go:64] FLAG: --version="false"
Jul 28 00:46:08.754326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745257    2564 flags.go:64] FLAG: --vmodule=""
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745263    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.745268    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745461    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745469    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745473    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745480    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745484    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745488    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745493    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745497    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745501    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745505    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745509    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745513    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745518    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745522    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745527    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745531    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745535    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:08.754921 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745539    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745543    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745547    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745552    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745555    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745559    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745563    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745567    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745571    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745575    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745578    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745584    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745588    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745594    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745599    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745603    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745610    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745615    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745622    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:08.755459 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745627    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745631    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745635    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745639    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745644    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745648    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745652    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745656    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745660    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745664    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745668    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745673    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745677    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745683    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745689    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745693    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745697    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745702    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745706    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745710    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:08.755948 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745714    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745719    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745723    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745728    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745733    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745737    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745741    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745746    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745750    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745754    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745758    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745764    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745768    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745772    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745776    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745781    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745785    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745791    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745796    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745801    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:08.756429 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745805    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745810    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745814    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745818    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745822    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745826    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745830    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745834    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745838    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745843    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745847    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745851    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745855    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745860    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745864    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745868    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745874    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745878    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745883    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:08.756919 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745887    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745892    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745897    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.745901    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.746876    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.754135    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.754151    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.754891    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.754900    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754975    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754983    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754987    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754990    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754992    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754995    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:08.757388 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.754998    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755001    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755004    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755007    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755009    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755012    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755015    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755017    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755020    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755022    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755025    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755027    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755030    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755033    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755035    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755038    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755040    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755045    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755048    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:08.757762 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755051    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755054    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755057    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755060    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755065    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755068    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755070    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755073    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755076    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755078    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755081    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755083    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755085    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755088    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755090    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755093    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755096    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755098    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755101    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:08.758246 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755103    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755106    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755108    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755111    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755113    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755116    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755118    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755121    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755124    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755126    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755129    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755131    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755134    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755136    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755139    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755141    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755144    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755147    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755150    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755153    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:08.758701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755155    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755158    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755160    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755163    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755166    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755168    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755171    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755173    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755176    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755178    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755181    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755183    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755186    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755188    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755191    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755193    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755196    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755199    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755201    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755205    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:08.759197 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755208    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755211    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755214    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755217    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755220    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755222    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755225    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755227    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755229    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755232    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755235    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755238    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755241    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755245    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755249    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:08.759672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.755254    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755406    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755411    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755415    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755418    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755421    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755423    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755426    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755429    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755432    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755435    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755437    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755440    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755443    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755446    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755450    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755453    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755456    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755458    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:08.760073 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755461    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755463    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755466    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755469    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755472    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755474    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755477    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755479    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755482    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755485    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755488    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755490    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755493    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755496    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755499    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755502    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755505    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755507    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755510    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755513    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:08.760527 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755517    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755520    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755523    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755526    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755529    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755532    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755534    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755537    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755540    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755543    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755545    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755548    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755550    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755553    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755555    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755558    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755560    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755563    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755565    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755567    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:08.761061 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755570    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755572    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755575    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755582    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755586    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755589    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755592    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755595    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755598    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755601    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755603    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755606    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755609    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755611    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755614    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755616    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755619    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755621    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755624    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755626    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:08.761517 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755629    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755632    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755634    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755636    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755639    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755642    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755644    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755647    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755649    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755651    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755654    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755657    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755659    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755662    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755664    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755667    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755675    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755678    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755680    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755683    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:08.762021 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:08.755686    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.755697    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.755845    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.758196    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.759478    2564 server.go:1019] "Starting client certificate rotation"
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.759604    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.759632    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 00:46:08.762485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.762211    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 00:46:08.766762 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.766740    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 00:46:08.779217 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.779196    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 28 00:46:08.781393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.781378    2564 log.go:25] "Validated CRI v1 image API"
Jul 28 00:46:08.783284 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.783270    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 00:46:08.786076 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.786048    2564 fs.go:135] Filesystem UUIDs: map[73f87ccb-65d4-4888-9de8-63b62f98870e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f0b472c1-9ab2-48d7-bc1b-a94efbe3e5a9:/dev/nvme0n1p4]
Jul 28 00:46:08.786160 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.786076    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 00:46:08.787902 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.787880    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 00:46:08.791892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.791777    2564 manager.go:216] Machine: {Timestamp:2026-07-28 00:46:08.790461003 +0000 UTC m=+0.345964173 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099096 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec242c22a225c84309cca9391721efd3 SystemUUID:ec242c22-a225-c843-09cc-a9391721efd3 BootID:aee4e4bd-2a89-473e-ac5f-9bf84a4bc36f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2d:cb:2e:c4:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2d:cb:2e:c4:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:f4:29:76:2c:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 00:46:08.791892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.791884    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 00:46:08.792065 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.792012    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 00:46:08.795381 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.795355    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 00:46:08.795518 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.795383    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 00:46:08.796802 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.796790    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 00:46:08.796835 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.796804    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 00:46:08.796835 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.796812    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 00:46:08.797499 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.797488    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 00:46:08.798320 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.798310    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 00:46:08.800005 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.799993    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 00:46:08.800114 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.800105    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 00:46:08.800711 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.800698    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 00:46:08.800711 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.800713    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 00:46:08.800797 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.800730    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 00:46:08.800797 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.800740    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 28 00:46:08.800797 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.800750    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 00:46:08.801090 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.801079    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 00:46:08.801131 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.801097    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 00:46:08.803602 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.803581    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 00:46:08.805616 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.805602    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 00:46:08.805685 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.805633    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 00:46:08.808502 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808486    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808508    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808515    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808521    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808529    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808534    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808542    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808547    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808554    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808560    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808568    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 00:46:08.808574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.808573    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 00:46:08.809645 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.809635    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 00:46:08.809645 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.809644    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 00:46:08.812707 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.812693    2564 server.go:1268] "Started kubelet"
Jul 28 00:46:08.813105 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.813089    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 00:46:08.813234 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.813216    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 00:46:08.813292 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.813279    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 00:46:08.813333 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.813313    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 00:46:08.813458 ip-10-0-131-111 systemd[1]: Started Kubernetes Kubelet.
Jul 28 00:46:08.815745 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.815723    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 00:46:08.816752 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.815765    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 00:46:08.818438 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.817540    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 00:46:08.818438 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.818190    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 00:46:08.823394 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.823367    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 00:46:08.824172 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.823062    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-111.ec2.internal.18c64d8ca84289df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-111.ec2.internal,UID:ip-10-0-131-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-111.ec2.internal,},FirstTimestamp:2026-07-28 00:46:08.812673503 +0000 UTC m=+0.368176654,LastTimestamp:2026-07-28 00:46:08.812673503 +0000 UTC m=+0.368176654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-111.ec2.internal,}"
Jul 28 00:46:08.826665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.826650    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 00:46:08.826759 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.826670    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 00:46:08.826823 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.826801    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:08.827353 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827328    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 00:46:08.827353 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827346    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 00:46:08.827494 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827330    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 00:46:08.827494 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827436    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 00:46:08.827494 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827444    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 00:46:08.827494 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827485    2564 factory.go:55] Registering systemd factory
Jul 28 00:46:08.827667 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.827499    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 28 00:46:08.828836 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.828622    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 00:46:08.828836 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.828710    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 00:46:08.828836 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.828785    2564 factory.go:153] Registering CRI-O factory
Jul 28 00:46:08.828836 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.828796    2564 factory.go:223] Registration of the crio container factory successfully
Jul 28 00:46:08.829015 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.828842    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 00:46:08.829015 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.828864    2564 factory.go:103] Registering Raw factory
Jul 28 00:46:08.829015 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.828877    2564 manager.go:1196] Started watching for new ooms in manager
Jul 28 00:46:08.829418 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.829259    2564 manager.go:349] Starting recovery of all containers
Jul 28 00:46:08.830580 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.830562    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kbs48"
Jul 28 00:46:08.837051 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.837019    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kbs48"
Jul 28 00:46:08.840684 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.840668    2564 manager.go:354] Recovery completed
Jul 28 00:46:08.845093 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.845055    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:08.847444 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.847425    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:08.847512 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.847463    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:08.847512 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.847477    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:08.848122 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.848104    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 00:46:08.848122 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.848122    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 00:46:08.848226 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.848139    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 00:46:08.848285 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.848267    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:08.849044 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.849027    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:08.849103 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.849054    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:08.849103 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.849067    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:08.849727 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.849713    2564 policy_none.go:49] "None policy: Start"
Jul 28 00:46:08.849727 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.849730    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 00:46:08.849829 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.849740    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 00:46:08.851033 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.851022    2564 policy_none.go:47] "Start"
Jul 28 00:46:08.889570 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.889550    2564 manager.go:337] "Starting Device Plugin manager"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.889585    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.889598    2564 server.go:87] "Starting device plugin registration server"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.889862    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.889873    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.890025    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.890137    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.890144    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.891733    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 00:46:08.900892 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.891795    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:08.937465 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.937442    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 00:46:08.938660 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.938645    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 00:46:08.938740 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.938686    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 00:46:08.938740 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.938712    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 00:46:08.938861 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:08.938750    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 00:46:08.941286 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.941267    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:08.990946 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.990880    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:08.994827 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.994808    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:08.994908 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.994842    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:08.994908 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.994855    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:08.994908 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:08.994878    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.003756 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.003740    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.003846 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.003764    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-111.ec2.internal\": node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.039179 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.039008    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"]
Jul 28 00:46:09.039354 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.039214    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:09.040794 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.040777    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:09.040898 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.040809    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:09.040898 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.040823    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:09.042308 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.042293    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:09.042416 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.042401    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.042485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.042441    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:09.042991 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.042974    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:09.042991 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.042986    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:09.043102 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.043004    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:09.043102 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.043003    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:09.043102 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.043015    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:09.043102 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.043026    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:09.044142 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.044129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.044205 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.044153    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:09.044782 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.044767    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:09.044862 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.044789    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:09.044862 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.044799    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:09.046706 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.046691    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.068619 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.068597    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-111.ec2.internal\" not found" node="ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.073124 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.073108    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-111.ec2.internal\" not found" node="ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.128658 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.128632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f04760098d23523919b1374e1df45b44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-111.ec2.internal\" (UID: \"f04760098d23523919b1374e1df45b44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.128658 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.128659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32c1bd98697f017a9de33b19186be610-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal\" (UID: \"32c1bd98697f017a9de33b19186be610\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.128813 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.128678    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c1bd98697f017a9de33b19186be610-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal\" (UID: \"32c1bd98697f017a9de33b19186be610\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.146829 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.146800    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.229217 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.229189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32c1bd98697f017a9de33b19186be610-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal\" (UID: \"32c1bd98697f017a9de33b19186be610\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.229363 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.229219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c1bd98697f017a9de33b19186be610-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal\" (UID: \"32c1bd98697f017a9de33b19186be610\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.229363 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.229238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f04760098d23523919b1374e1df45b44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-111.ec2.internal\" (UID: \"f04760098d23523919b1374e1df45b44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.229363 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.229283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f04760098d23523919b1374e1df45b44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-111.ec2.internal\" (UID: \"f04760098d23523919b1374e1df45b44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.229363 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.229308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32c1bd98697f017a9de33b19186be610-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal\" (UID: \"32c1bd98697f017a9de33b19186be610\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.229363 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.229325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c1bd98697f017a9de33b19186be610-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal\" (UID: \"32c1bd98697f017a9de33b19186be610\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.247328 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.247262    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.348128 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.348094    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.371300 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.371272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.376560 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.376341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:09.448549 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.448514    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.548798 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.548762    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.649256 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.649216    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.691656 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.691634    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:09.749840 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.749803    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.758584 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.758561    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 00:46:09.758702 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.758685    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 00:46:09.758741 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.758723    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 00:46:09.842646 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.842554    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 00:41:09 +0000 UTC" deadline="2028-02-18 05:38:58.300572983 +0000 UTC"
Jul 28 00:46:09.842646 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:09.842586    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13684h52m48.457990066s"
Jul 28 00:46:09.850823 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.850802    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:09.951078 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:09.951049    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:10.052172 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.052142    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-111.ec2.internal\" not found"
Jul 28 00:46:10.055575 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:10.055543    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c1bd98697f017a9de33b19186be610.slice/crio-6c682d9c394b5bfd735b514d87356d61e984d004912e6a4cbe646930b51c1bfb WatchSource:0}: Error finding container 6c682d9c394b5bfd735b514d87356d61e984d004912e6a4cbe646930b51c1bfb: Status 404 returned error can't find the container with id 6c682d9c394b5bfd735b514d87356d61e984d004912e6a4cbe646930b51c1bfb
Jul 28 00:46:10.060530 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.060511    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 00:46:10.061191 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.061175    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:10.127427 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.127334    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:10.143551 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.143521    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 00:46:10.145481 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.145466    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal"
Jul 28 00:46:10.156206 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.156183    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 00:46:10.272199 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.272159    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:10.465386 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:10.465353    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04760098d23523919b1374e1df45b44.slice/crio-e43d27495b308368e78b867aaac28a4acd95a6d18b1e05adafbf4a1233818218 WatchSource:0}: Error finding container e43d27495b308368e78b867aaac28a4acd95a6d18b1e05adafbf4a1233818218: Status 404 returned error can't find the container with id e43d27495b308368e78b867aaac28a4acd95a6d18b1e05adafbf4a1233818218
Jul 28 00:46:10.794903 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.794824    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:10.803108 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.803085    2564 apiserver.go:52] "Watching apiserver"
Jul 28 00:46:10.808265 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.808242    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 00:46:10.811290 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.811267    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal","openshift-multus/network-metrics-daemon-g88nz","openshift-network-operator/iptables-alerter-pdnsv","openshift-ovn-kubernetes/ovnkube-node-rsmjh","kube-system/global-pull-secret-syncer-6vvw7","kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mrm5w","openshift-dns/node-resolver-66whq","openshift-image-registry/node-ca-gf5ph","openshift-multus/multus-additional-cni-plugins-jrzvt","openshift-multus/multus-vwp85","openshift-network-diagnostics/network-check-target-z5mrk","kube-system/konnectivity-agent-qtd5t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"]
Jul 28 00:46:10.813052 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.813028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.814442 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.814353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.815152 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.815130    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fdn7s\""
Jul 28 00:46:10.815275 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.815257    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.815703 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.815536    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.815853 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.815832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.816780 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.816667    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 00:46:10.816780 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.816770    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-29xrf\""
Jul 28 00:46:10.816924 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.816911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 00:46:10.817657 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.817638    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.818433 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.818243    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.818433 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.818385    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 00:46:10.818599 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.818521    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4dgnx\""
Jul 28 00:46:10.818599 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.818581    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.819506 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.819285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.819506 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.819375    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:10.820686 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.820498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 00:46:10.820783 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.820692    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.820848 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.820799    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 00:46:10.821235 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.821173    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 00:46:10.822324 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.821710    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.822324 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.821795    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.822324 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.822224    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 00:46:10.822563 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.822508    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvglm\""
Jul 28 00:46:10.823857 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.823694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.825548 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.825527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 00:46:10.825704 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.825685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.825804 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.825780    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.826048 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.826026    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 00:46:10.826163 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.826143    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.826231 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.826030    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.826296 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.826275    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5ct4w\""
Jul 28 00:46:10.826411 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.826393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bzjmc\""
Jul 28 00:46:10.826488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.826470    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.827262 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.827214    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 00:46:10.827817 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.827217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:10.827901 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.827876    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:10.828003 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.827981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xt8b5\""
Jul 28 00:46:10.828385 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.828366    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.828460 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.828420    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.829048 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.829028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.830856 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.830837    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.831505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.831482    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 00:46:10.831605 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.831533    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 00:46:10.831605 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.831546    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 00:46:10.831605 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.831583    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 00:46:10.831881 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.831862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nskxd\""
Jul 28 00:46:10.832009 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.831873    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 00:46:10.832505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.832485    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:10.832597 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.832551    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:10.833232 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.833210    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 00:46:10.833337 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.833281    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vzxbh\""
Jul 28 00:46:10.838893 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.838872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-kubelet\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839003 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.838909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a848cccd-6c9b-433f-8554-95b82faec0e6-ovn-node-metrics-cert\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839003 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.838935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b423eba2-2ccd-469a-99a2-fca1cc47a898-host-slash\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.839003 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.838974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31c2a323-206c-4f9c-b65c-555120aa0507-host\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.839182 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-systemd\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839182 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-lib-modules\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839182 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc47e71c-d9f7-4775-a22b-ae0f97724419-tmp\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839182 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-systemd\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839182 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-system-cni-dir\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-sys\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/580545c3-37c3-4853-89e3-3b35c757b895-agent-certs\") pod \"konnectivity-agent-qtd5t\" (UID: \"580545c3-37c3-4853-89e3-3b35c757b895\") " pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b423eba2-2ccd-469a-99a2-fca1cc47a898-iptables-alerter-script\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-kubernetes\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysctl-d\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-var-lib-kubelet\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-run-netns\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839478    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8lh2\" (UniqueName: \"kubernetes.io/projected/b423eba2-2ccd-469a-99a2-fca1cc47a898-kube-api-access-h8lh2\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-tuned\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/580545c3-37c3-4853-89e3-3b35c757b895-konnectivity-ca\") pod \"konnectivity-agent-qtd5t\" (UID: \"580545c3-37c3-4853-89e3-3b35c757b895\") " pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-ovnkube-script-lib\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn5v\" (UniqueName: \"kubernetes.io/projected/c30d7d6a-eb19-4e83-aaf6-63edaccece49-kube-api-access-6sn5v\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31c2a323-206c-4f9c-b65c-555120aa0507-serviceca\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839719    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.839758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-etc-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-ovnkube-config\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839826    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839859    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-sys-fs\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp6s\" (UniqueName: \"kubernetes.io/projected/fc47e71c-d9f7-4775-a22b-ae0f97724419-kube-api-access-qpp6s\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-cni-bin\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839972    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-env-overrides\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.839998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-cnibin\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-socket-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840061    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6zr\" (UniqueName: \"kubernetes.io/projected/31c2a323-206c-4f9c-b65c-555120aa0507-kube-api-access-fh6zr\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysconfig\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-node-log\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840193    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwns9\" (UniqueName: \"kubernetes.io/projected/a339eb81-8e58-4560-8c0b-d37f7750813e-kube-api-access-dwns9\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-run\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840224    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 00:46:10.840331 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-slash\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-var-lib-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840298    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-cni-netd\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-os-release\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef6df0-7787-4f83-8768-06920413b08a-tmp-dir\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-systemd-units\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840549    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-kubelet-config\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-dbus\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-registration-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-etc-selinux\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-ovn\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-log-socket\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9xt\" (UniqueName: \"kubernetes.io/projected/a848cccd-6c9b-433f-8554-95b82faec0e6-kube-api-access-9q9xt\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840828    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-device-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.840988 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfef6df0-7787-4f83-8768-06920413b08a-hosts-file\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.841744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96vr\" (UniqueName: \"kubernetes.io/projected/cfef6df0-7787-4f83-8768-06920413b08a-kube-api-access-h96vr\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.841744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7sg9\" (UniqueName: \"kubernetes.io/projected/0da18fd8-305e-4996-a531-2f569665d052-kube-api-access-p7sg9\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:10.841744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-modprobe-d\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.841744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:10.841744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.840998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysctl-conf\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.841744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.841025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-host\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.862615 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.862590    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2rvjl"
Jul 28 00:46:10.870728 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.870704    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2rvjl"
Jul 28 00:46:10.928283 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.928256    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 00:46:10.941419 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/580545c3-37c3-4853-89e3-3b35c757b895-konnectivity-ca\") pod \"konnectivity-agent-qtd5t\" (UID: \"580545c3-37c3-4853-89e3-3b35c757b895\") " pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.941419 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-ovnkube-script-lib\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn5v\" (UniqueName: \"kubernetes.io/projected/c30d7d6a-eb19-4e83-aaf6-63edaccece49-kube-api-access-6sn5v\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31c2a323-206c-4f9c-b65c-555120aa0507-serviceca\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-system-cni-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-etc-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-ovnkube-config\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.941633 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-sys-fs\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp6s\" (UniqueName: \"kubernetes.io/projected/fc47e71c-d9f7-4775-a22b-ae0f97724419-kube-api-access-qpp6s\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-cni-multus\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-cni-bin\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-env-overrides\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-cnibin\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-socket-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6zr\" (UniqueName: \"kubernetes.io/projected/31c2a323-206c-4f9c-b65c-555120aa0507-kube-api-access-fh6zr\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysconfig\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-cnibin\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-netns\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-node-log\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.941990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwns9\" (UniqueName: \"kubernetes.io/projected/a339eb81-8e58-4560-8c0b-d37f7750813e-kube-api-access-dwns9\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-run\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.942047 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-cni-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-kubelet\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-slash\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/580545c3-37c3-4853-89e3-3b35c757b895-konnectivity-ca\") pod \"konnectivity-agent-qtd5t\" (UID: \"580545c3-37c3-4853-89e3-3b35c757b895\") " pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-var-lib-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-cni-netd\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-os-release\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-cni-netd\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-cnibin\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef6df0-7787-4f83-8768-06920413b08a-tmp-dir\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-k8s-cni-cncf-io\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-conf-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-systemd-units\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-socket-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.942831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-kubelet-config\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-env-overrides\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-dbus\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-registration-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-etc-selinux\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-ovn\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-log-socket\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-ovnkube-script-lib\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9xt\" (UniqueName: \"kubernetes.io/projected/a848cccd-6c9b-433f-8554-95b82faec0e6-kube-api-access-9q9xt\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-device-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-node-log\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfef6df0-7787-4f83-8768-06920413b08a-hosts-file\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96vr\" (UniqueName: \"kubernetes.io/projected/cfef6df0-7787-4f83-8768-06920413b08a-kube-api-access-h96vr\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-var-lib-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-dbus\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-slash\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-ovn\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.943700 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-cni-bin\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942946    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-registration-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.942781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7sg9\" (UniqueName: \"kubernetes.io/projected/0da18fd8-305e-4996-a531-2f569665d052-kube-api-access-p7sg9\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef6df0-7787-4f83-8768-06920413b08a-tmp-dir\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-etc-selinux\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-os-release\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-run\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-systemd-units\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a848cccd-6c9b-433f-8554-95b82faec0e6-ovnkube-config\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-device-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysconfig\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-modprobe-d\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943315    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfef6df0-7787-4f83-8768-06920413b08a-hosts-file\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-kubelet-config\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysctl-conf\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.944486 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-sys-fs\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-host\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.943397    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943532    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-socket-dir-parent\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-log-socket\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-cni-bin\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31c2a323-206c-4f9c-b65c-555120aa0507-serviceca\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-hostroot\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a339eb81-8e58-4560-8c0b-d37f7750813e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-host\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-multus-certs\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnz86\" (UniqueName: \"kubernetes.io/projected/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-kube-api-access-wnz86\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943664    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c30d7d6a-eb19-4e83-aaf6-63edaccece49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-modprobe-d\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-etc-kubernetes\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.945309 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysctl-conf\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.943737    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:11.443714977 +0000 UTC m=+2.999218162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-kubelet\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.943839    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a848cccd-6c9b-433f-8554-95b82faec0e6-ovn-node-metrics-cert\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-etc-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:10.943886    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:11.443871866 +0000 UTC m=+2.999375013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b423eba2-2ccd-469a-99a2-fca1cc47a898-host-slash\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943925    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-kubelet\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31c2a323-206c-4f9c-b65c-555120aa0507-host\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.943989    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b423eba2-2ccd-469a-99a2-fca1cc47a898-host-slash\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-systemd\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31c2a323-206c-4f9c-b65c-555120aa0507-host\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944122    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-systemd\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-lib-modules\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc47e71c-d9f7-4775-a22b-ae0f97724419-tmp\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-systemd\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.946194 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944213    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-lib-modules\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-system-cni-dir\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-systemd\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-sys\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-sys\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-cni-binary-copy\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/580545c3-37c3-4853-89e3-3b35c757b895-agent-certs\") pod \"konnectivity-agent-qtd5t\" (UID: \"580545c3-37c3-4853-89e3-3b35c757b895\") " pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b423eba2-2ccd-469a-99a2-fca1cc47a898-iptables-alerter-script\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-system-cni-dir\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-kubernetes\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a339eb81-8e58-4560-8c0b-d37f7750813e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysctl-d\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944789    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-kubernetes\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-var-lib-kubelet\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-sysctl-d\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944848    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-os-release\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-run-netns\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc47e71c-d9f7-4775-a22b-ae0f97724419-var-lib-kubelet\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.944998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-run-openvswitch\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.945001    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8lh2\" (UniqueName: \"kubernetes.io/projected/b423eba2-2ccd-469a-99a2-fca1cc47a898-kube-api-access-h8lh2\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.945002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-run-netns\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.945033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-tuned\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.945037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a848cccd-6c9b-433f-8554-95b82faec0e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.945069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-daemon-config\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.945265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b423eba2-2ccd-469a-99a2-fca1cc47a898-iptables-alerter-script\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:10.947984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.946893    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal" event={"ID":"f04760098d23523919b1374e1df45b44","Type":"ContainerStarted","Data":"e43d27495b308368e78b867aaac28a4acd95a6d18b1e05adafbf4a1233818218"}
Jul 28 00:46:10.948657 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.948414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc47e71c-d9f7-4775-a22b-ae0f97724419-tmp\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.948657 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.948448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a848cccd-6c9b-433f-8554-95b82faec0e6-ovn-node-metrics-cert\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.948657 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.948638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/580545c3-37c3-4853-89e3-3b35c757b895-agent-certs\") pod \"konnectivity-agent-qtd5t\" (UID: \"580545c3-37c3-4853-89e3-3b35c757b895\") " pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:10.948869 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.948848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc47e71c-d9f7-4775-a22b-ae0f97724419-etc-tuned\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.949071 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.948978    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal" event={"ID":"32c1bd98697f017a9de33b19186be610","Type":"ContainerStarted","Data":"6c682d9c394b5bfd735b514d87356d61e984d004912e6a4cbe646930b51c1bfb"}
Jul 28 00:46:10.950513 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.950485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6zr\" (UniqueName: \"kubernetes.io/projected/31c2a323-206c-4f9c-b65c-555120aa0507-kube-api-access-fh6zr\") pod \"node-ca-gf5ph\" (UID: \"31c2a323-206c-4f9c-b65c-555120aa0507\") " pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:10.950773 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.950751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwns9\" (UniqueName: \"kubernetes.io/projected/a339eb81-8e58-4560-8c0b-d37f7750813e-kube-api-access-dwns9\") pod \"multus-additional-cni-plugins-jrzvt\" (UID: \"a339eb81-8e58-4560-8c0b-d37f7750813e\") " pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:10.951289 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.951270    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9xt\" (UniqueName: \"kubernetes.io/projected/a848cccd-6c9b-433f-8554-95b82faec0e6-kube-api-access-9q9xt\") pod \"ovnkube-node-rsmjh\" (UID: \"a848cccd-6c9b-433f-8554-95b82faec0e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:10.954100 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.954068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn5v\" (UniqueName: \"kubernetes.io/projected/c30d7d6a-eb19-4e83-aaf6-63edaccece49-kube-api-access-6sn5v\") pod \"aws-ebs-csi-driver-node-7m6bz\" (UID: \"c30d7d6a-eb19-4e83-aaf6-63edaccece49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:10.957927 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.957876    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7sg9\" (UniqueName: \"kubernetes.io/projected/0da18fd8-305e-4996-a531-2f569665d052-kube-api-access-p7sg9\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:10.958296 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.958273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp6s\" (UniqueName: \"kubernetes.io/projected/fc47e71c-d9f7-4775-a22b-ae0f97724419-kube-api-access-qpp6s\") pod \"tuned-mrm5w\" (UID: \"fc47e71c-d9f7-4775-a22b-ae0f97724419\") " pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:10.958668 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.958648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96vr\" (UniqueName: \"kubernetes.io/projected/cfef6df0-7787-4f83-8768-06920413b08a-kube-api-access-h96vr\") pod \"node-resolver-66whq\" (UID: \"cfef6df0-7787-4f83-8768-06920413b08a\") " pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:10.961048 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:10.961023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8lh2\" (UniqueName: \"kubernetes.io/projected/b423eba2-2ccd-469a-99a2-fca1cc47a898-kube-api-access-h8lh2\") pod \"iptables-alerter-pdnsv\" (UID: \"b423eba2-2ccd-469a-99a2-fca1cc47a898\") " pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:11.046381 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-system-cni-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046381 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-cni-multus\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046381 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-cnibin\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-system-cni-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046434    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-cni-multus\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-netns\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-netns\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046493    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-cnibin\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-cni-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-kubelet\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-cni-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-k8s-cni-cncf-io\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-conf-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-kubelet\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.046665 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-conf-dir\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-k8s-cni-cncf-io\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-socket-dir-parent\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-cni-bin\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-hostroot\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-multus-certs\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-socket-dir-parent\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnz86\" (UniqueName: \"kubernetes.io/projected/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-kube-api-access-wnz86\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.046959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-etc-kubernetes\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-cni-binary-copy\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-os-release\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-daemon-config\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-var-lib-cni-bin\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-hostroot\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.047393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-host-run-multus-certs\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.048051 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-multus-daemon-config\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.048051 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-cni-binary-copy\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.048051 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-etc-kubernetes\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.048051 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.047852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-os-release\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.052842 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.052813    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:11.052842 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.052840    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:11.053053 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.052855    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:11.053053 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.052924    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:46:11.552906363 +0000 UTC m=+3.108409505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:11.055418 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.055394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnz86\" (UniqueName: \"kubernetes.io/projected/9d258fb9-c885-41b7-88d1-93fe25e3c4eb-kube-api-access-wnz86\") pod \"multus-vwp85\" (UID: \"9d258fb9-c885-41b7-88d1-93fe25e3c4eb\") " pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.127987 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.127920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrm5w"
Jul 28 00:46:11.135083 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.134999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:11.140159 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.140128    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc47e71c_d9f7_4775_a22b_ae0f97724419.slice/crio-0f973df6ab57b58fa2fdb12447c8a08b3e414cff2779dcf82c8ec582678beb0d WatchSource:0}: Error finding container 0f973df6ab57b58fa2fdb12447c8a08b3e414cff2779dcf82c8ec582678beb0d: Status 404 returned error can't find the container with id 0f973df6ab57b58fa2fdb12447c8a08b3e414cff2779dcf82c8ec582678beb0d
Jul 28 00:46:11.143898 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.143877    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:11.150390 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.150365    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda848cccd_6c9b_433f_8554_95b82faec0e6.slice/crio-30fd9ce6dacfc38c4f7d126f3b76732aac0e3f649a84ae7f308cb5c0f9bdde6b WatchSource:0}: Error finding container 30fd9ce6dacfc38c4f7d126f3b76732aac0e3f649a84ae7f308cb5c0f9bdde6b: Status 404 returned error can't find the container with id 30fd9ce6dacfc38c4f7d126f3b76732aac0e3f649a84ae7f308cb5c0f9bdde6b
Jul 28 00:46:11.151503 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.151474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdnsv"
Jul 28 00:46:11.158751 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.158719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz"
Jul 28 00:46:11.159891 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.159806    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580545c3_37c3_4853_89e3_3b35c757b895.slice/crio-3625369a1a034f34857441ea7aca1d323b4478d6148fe6e45013f605606d7c27 WatchSource:0}: Error finding container 3625369a1a034f34857441ea7aca1d323b4478d6148fe6e45013f605606d7c27: Status 404 returned error can't find the container with id 3625369a1a034f34857441ea7aca1d323b4478d6148fe6e45013f605606d7c27
Jul 28 00:46:11.164199 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.164178    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb423eba2_2ccd_469a_99a2_fca1cc47a898.slice/crio-686d988d7aba2c2d6559cf497c719996b051c463b8d547557d3bca751450f742 WatchSource:0}: Error finding container 686d988d7aba2c2d6559cf497c719996b051c463b8d547557d3bca751450f742: Status 404 returned error can't find the container with id 686d988d7aba2c2d6559cf497c719996b051c463b8d547557d3bca751450f742
Jul 28 00:46:11.165420 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.165255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gf5ph"
Jul 28 00:46:11.172204 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.172181    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66whq"
Jul 28 00:46:11.174839 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.174813    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30d7d6a_eb19_4e83_aaf6_63edaccece49.slice/crio-e897efa00d55089866d12e2ed57a5d14248872c55814b39b1215077feaa26b56 WatchSource:0}: Error finding container e897efa00d55089866d12e2ed57a5d14248872c55814b39b1215077feaa26b56: Status 404 returned error can't find the container with id e897efa00d55089866d12e2ed57a5d14248872c55814b39b1215077feaa26b56
Jul 28 00:46:11.180095 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.180072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrzvt"
Jul 28 00:46:11.187739 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.187537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vwp85"
Jul 28 00:46:11.188078 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.188032    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c2a323_206c_4f9c_b65c_555120aa0507.slice/crio-2f287eeaf2ca00e816f5046c428d3aaf73ba417c15472cb549466b888670895b WatchSource:0}: Error finding container 2f287eeaf2ca00e816f5046c428d3aaf73ba417c15472cb549466b888670895b: Status 404 returned error can't find the container with id 2f287eeaf2ca00e816f5046c428d3aaf73ba417c15472cb549466b888670895b
Jul 28 00:46:11.197094 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.197062    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda339eb81_8e58_4560_8c0b_d37f7750813e.slice/crio-0f4b78348121bec80c96f565020d5148a4e6b071b720157e012c6db91d8a35d4 WatchSource:0}: Error finding container 0f4b78348121bec80c96f565020d5148a4e6b071b720157e012c6db91d8a35d4: Status 404 returned error can't find the container with id 0f4b78348121bec80c96f565020d5148a4e6b071b720157e012c6db91d8a35d4
Jul 28 00:46:11.200052 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.199744    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfef6df0_7787_4f83_8768_06920413b08a.slice/crio-69e18786e2db4e7aba63e867c2c69d7a4bf4418e8c3fdd92c47e0c4a92b38e1b WatchSource:0}: Error finding container 69e18786e2db4e7aba63e867c2c69d7a4bf4418e8c3fdd92c47e0c4a92b38e1b: Status 404 returned error can't find the container with id 69e18786e2db4e7aba63e867c2c69d7a4bf4418e8c3fdd92c47e0c4a92b38e1b
Jul 28 00:46:11.205475 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:11.205449    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d258fb9_c885_41b7_88d1_93fe25e3c4eb.slice/crio-45bdcaeff6bea248eb0c0e0ba2f6864e2833d0a552867c2a045b30f265fbcd7f WatchSource:0}: Error finding container 45bdcaeff6bea248eb0c0e0ba2f6864e2833d0a552867c2a045b30f265fbcd7f: Status 404 returned error can't find the container with id 45bdcaeff6bea248eb0c0e0ba2f6864e2833d0a552867c2a045b30f265fbcd7f
Jul 28 00:46:11.450682 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.450590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:11.450682 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.450668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:11.451002 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.450776    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:11.451002 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.450885    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:12.450862805 +0000 UTC m=+4.006365965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:11.451002 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.450788    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:11.451002 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.450978    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:12.450959007 +0000 UTC m=+4.006462155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:11.652229 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.652194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:11.652399 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.652380    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:11.652446 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.652403    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:11.652446 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.652414    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:11.652519 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:11.652479    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:46:12.652463218 +0000 UTC m=+4.207966372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:11.662108 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.662069    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:11.873071 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.872836    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:41:11 +0000 UTC" deadline="2028-02-06 04:47:00.068340907 +0000 UTC"
Jul 28 00:46:11.873071 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.872917    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13396h0m48.195429138s"
Jul 28 00:46:11.953634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.953593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qtd5t" event={"ID":"580545c3-37c3-4853-89e3-3b35c757b895","Type":"ContainerStarted","Data":"3625369a1a034f34857441ea7aca1d323b4478d6148fe6e45013f605606d7c27"}
Jul 28 00:46:11.958287 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.958254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrm5w" event={"ID":"fc47e71c-d9f7-4775-a22b-ae0f97724419","Type":"ContainerStarted","Data":"0f973df6ab57b58fa2fdb12447c8a08b3e414cff2779dcf82c8ec582678beb0d"}
Jul 28 00:46:11.962696 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.962660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66whq" event={"ID":"cfef6df0-7787-4f83-8768-06920413b08a","Type":"ContainerStarted","Data":"69e18786e2db4e7aba63e867c2c69d7a4bf4418e8c3fdd92c47e0c4a92b38e1b"}
Jul 28 00:46:11.966696 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.966661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gf5ph" event={"ID":"31c2a323-206c-4f9c-b65c-555120aa0507","Type":"ContainerStarted","Data":"2f287eeaf2ca00e816f5046c428d3aaf73ba417c15472cb549466b888670895b"}
Jul 28 00:46:11.968766 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.968738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdnsv" event={"ID":"b423eba2-2ccd-469a-99a2-fca1cc47a898","Type":"ContainerStarted","Data":"686d988d7aba2c2d6559cf497c719996b051c463b8d547557d3bca751450f742"}
Jul 28 00:46:11.971980 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.971938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"30fd9ce6dacfc38c4f7d126f3b76732aac0e3f649a84ae7f308cb5c0f9bdde6b"}
Jul 28 00:46:11.973705 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.973623    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vwp85" event={"ID":"9d258fb9-c885-41b7-88d1-93fe25e3c4eb","Type":"ContainerStarted","Data":"45bdcaeff6bea248eb0c0e0ba2f6864e2833d0a552867c2a045b30f265fbcd7f"}
Jul 28 00:46:11.978718 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.978689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerStarted","Data":"0f4b78348121bec80c96f565020d5148a4e6b071b720157e012c6db91d8a35d4"}
Jul 28 00:46:11.981771 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:11.981742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz" event={"ID":"c30d7d6a-eb19-4e83-aaf6-63edaccece49","Type":"ContainerStarted","Data":"e897efa00d55089866d12e2ed57a5d14248872c55814b39b1215077feaa26b56"}
Jul 28 00:46:12.459222 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.458383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:12.459222 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.458456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:12.459222 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.458622    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:12.459222 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.458689    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:14.458669251 +0000 UTC m=+6.014172394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:12.459222 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.459129    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:12.459222 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.459184    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:14.459167739 +0000 UTC m=+6.014670882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:12.660061 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.660020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:12.660282 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.660262    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:12.660355 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.660294    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:12.660355 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.660307    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:12.660461 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.660369    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:46:14.660349483 +0000 UTC m=+6.215852629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:12.873278 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.873170    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:41:11 +0000 UTC" deadline="2028-03-16 05:56:05.150756813 +0000 UTC"
Jul 28 00:46:12.873278 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.873216    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14333h9m52.277545833s"
Jul 28 00:46:12.942082 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.942049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:12.942257 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.942195    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:12.942638 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.942616    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:12.942750 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.942731    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:12.942816 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:12.942795    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:12.942882 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:12.942861    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:14.004101 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.003829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal" event={"ID":"f04760098d23523919b1374e1df45b44","Type":"ContainerStarted","Data":"9f4fb5678f1ae8451f0c6c01010cc6b1403694ba056984404db7e23ee257b738"}
Jul 28 00:46:14.023136 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.023077    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-111.ec2.internal" podStartSLOduration=4.023056855 podStartE2EDuration="4.023056855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:14.022842169 +0000 UTC m=+5.578345332" watchObservedRunningTime="2026-07-28 00:46:14.023056855 +0000 UTC m=+5.578560019"
Jul 28 00:46:14.475536 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.475151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:14.475536 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.475264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:14.475536 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.475312    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:14.475536 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.475394    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:18.475369918 +0000 UTC m=+10.030873065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:14.475536 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.475432    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:14.475536 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.475483    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:18.475467271 +0000 UTC m=+10.030970411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:14.677243 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.676633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:14.677243 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.676786    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:14.677243 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.676807    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:14.677243 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.676829    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:14.677243 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.676887    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:46:18.676869354 +0000 UTC m=+10.232372506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:14.940170 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.939444    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:14.940170 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.939562    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:14.940170 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.939951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:14.940170 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.940058    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:14.940657 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:14.940532    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:14.940657 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:14.940624    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:15.023887 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:15.023804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal" event={"ID":"32c1bd98697f017a9de33b19186be610","Type":"ContainerStarted","Data":"6ce5f0646450102d441b08eaaf9b06df08a04d3a8057688c98c466ebfa2d4a0e"}
Jul 28 00:46:16.027276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:16.027053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal" event={"ID":"32c1bd98697f017a9de33b19186be610","Type":"ContainerDied","Data":"6ce5f0646450102d441b08eaaf9b06df08a04d3a8057688c98c466ebfa2d4a0e"}
Jul 28 00:46:16.027731 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:16.026927    2564 generic.go:356] "Generic (PLEG): container finished" podID="32c1bd98697f017a9de33b19186be610" containerID="6ce5f0646450102d441b08eaaf9b06df08a04d3a8057688c98c466ebfa2d4a0e" exitCode=0
Jul 28 00:46:16.939755 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:16.939723    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:16.939977 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:16.939865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:16.940454 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:16.940307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:16.940454 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:16.940406    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:16.940843 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:16.940712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:16.940843 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:16.940810    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:18.508004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:18.507387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:18.508004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:18.507471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:18.508004 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.507599    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:18.508004 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.507659    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:26.50763889 +0000 UTC m=+18.063142031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:18.508004 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.507808    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:18.508004 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.507875    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:26.507858267 +0000 UTC m=+18.063361422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:18.709662 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:18.709505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:18.709841 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.709706    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:18.709841 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.709727    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:18.709841 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.709738    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:18.709841 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.709796    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:46:26.709777924 +0000 UTC m=+18.265281062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:18.940172 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:18.940088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:18.940318 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.940203    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:18.940318 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:18.940263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:18.940435 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.940339    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:18.940892 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:18.940675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:18.940892 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:18.940769    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:20.939927 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:20.939894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:20.940364 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:20.939894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:20.940364 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:20.939903    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:20.940364 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:20.940039    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:20.940364 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:20.940173    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:20.940364 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:20.940261    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:22.939457 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:22.939363    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:22.939888 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:22.939516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:22.939888 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:22.939525    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:22.939888 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:22.939638    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:22.939888 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:22.939685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:22.939888 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:22.939795    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:24.939376 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:24.939340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:24.939870 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:24.939341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:24.939870 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:24.939466    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:24.939870 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:24.939567    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:24.939870 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:24.939341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:24.939870 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:24.939649    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:26.570113 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:26.570077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:26.570601 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:26.570165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:26.570601 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.570276    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:26.570601 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.570290    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:26.570601 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.570351    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:42.570335263 +0000 UTC m=+34.125838401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:26.570601 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.570367    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:42.570360456 +0000 UTC m=+34.125863593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:26.771504 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:26.771461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:26.771694 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.771657    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:26.771694 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.771688    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:26.771812 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.771700    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:26.771812 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.771767    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:46:42.771748578 +0000 UTC m=+34.327251719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:26.938994 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:26.938901    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:26.939147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:26.938901    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:26.939147 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.939039    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:26.939234 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.939150    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:26.939234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:26.938905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:26.939361 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:26.939245    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:28.940994 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:28.940957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:28.941551 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:28.941016    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:28.941551 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:28.941108    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:28.941551 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:28.941127    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:28.941551 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:28.941149    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:28.941551 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:28.941235    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:30.939729 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:30.939304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:30.940722 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:30.939317    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:30.940722 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:30.939847    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:30.940722 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:30.939330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:30.940722 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:30.939927    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:30.940722 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:30.940317    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:31.063383 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.063355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"e242422e8f2941da8d23ef9b004bd6e00d3c9358307f6bd0090a9449ae8e51c6"}
Jul 28 00:46:31.063530 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.063429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"82c12be032fa15b4f685515c38cb6d533f0188d0f8daef9e99eae6e51a5bf501"}
Jul 28 00:46:31.063530 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.063473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"1287c82d559db2aa5edd55f01ff5b8aeed26ed2ecd52bd6e3d362e2036b6bffc"}
Jul 28 00:46:31.064919 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.064895    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal" event={"ID":"32c1bd98697f017a9de33b19186be610","Type":"ContainerStarted","Data":"ffa659bc98c552b9f34c8a47f5af0bc5aa8e863e1ec0b4f913eacaf951579a41"}
Jul 28 00:46:31.066225 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.066203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vwp85" event={"ID":"9d258fb9-c885-41b7-88d1-93fe25e3c4eb","Type":"ContainerStarted","Data":"2c8a8e259a283dd9a66e58654450fc25b1cbc1b2bbba8fcd467e2a5c6ea51403"}
Jul 28 00:46:31.067834 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.067813    2564 generic.go:356] "Generic (PLEG): container finished" podID="a339eb81-8e58-4560-8c0b-d37f7750813e" containerID="1cf2b7085b1eaad4efdb2fb5a83f2fe31954d178b7f87d88e0ba0b70a03a5787" exitCode=0
Jul 28 00:46:31.067921 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.067879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerDied","Data":"1cf2b7085b1eaad4efdb2fb5a83f2fe31954d178b7f87d88e0ba0b70a03a5787"}
Jul 28 00:46:31.069408 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.069380    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz" event={"ID":"c30d7d6a-eb19-4e83-aaf6-63edaccece49","Type":"ContainerStarted","Data":"bd59b03eb7c6f05db2375ff82ef454c4acd94ea56512faacfed79aea45a63b50"}
Jul 28 00:46:31.072626 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.072597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qtd5t" event={"ID":"580545c3-37c3-4853-89e3-3b35c757b895","Type":"ContainerStarted","Data":"1e688589146eba65d8923426ea05695fb9e42f827ec2bd0522b837affaec9284"}
Jul 28 00:46:31.073976 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.073953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrm5w" event={"ID":"fc47e71c-d9f7-4775-a22b-ae0f97724419","Type":"ContainerStarted","Data":"581e23d5952dfd54af74e6143aab8faef8b283025e31047dc10c010ce9f8e814"}
Jul 28 00:46:31.075286 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.075266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66whq" event={"ID":"cfef6df0-7787-4f83-8768-06920413b08a","Type":"ContainerStarted","Data":"b7ac97a18a2ce159fef36425d34a7b1315361f15bea6c236d8210ab8ed017544"}
Jul 28 00:46:31.076641 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.076617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gf5ph" event={"ID":"31c2a323-206c-4f9c-b65c-555120aa0507","Type":"ContainerStarted","Data":"61401857b651934b0e5a80293192a1f60b223f45287f1307761fee5cc04535d9"}
Jul 28 00:46:31.078860 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.078806    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-111.ec2.internal" podStartSLOduration=21.07878896 podStartE2EDuration="21.07878896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:31.078130854 +0000 UTC m=+22.633634013" watchObservedRunningTime="2026-07-28 00:46:31.07878896 +0000 UTC m=+22.634292123"
Jul 28 00:46:31.093572 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.093532    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-66whq" podStartSLOduration=2.789092275 podStartE2EDuration="22.093517208s" totalImagesPullingTime="19.304424933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:31.092829764 +0000 UTC m=+22.648332921" watchObservedRunningTime="2026-07-28 00:46:31.093517208 +0000 UTC m=+22.649020368"
Jul 28 00:46:31.108197 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.108155    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vwp85" podStartSLOduration=2.757261591 podStartE2EDuration="22.108140865s" totalImagesPullingTime="19.350879274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:31.108114741 +0000 UTC m=+22.663617902" watchObservedRunningTime="2026-07-28 00:46:31.108140865 +0000 UTC m=+22.663644003"
Jul 28 00:46:31.123414 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.123373    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mrm5w" podStartSLOduration=2.729608567 podStartE2EDuration="22.123358699s" totalImagesPullingTime="19.393750132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:31.122751645 +0000 UTC m=+22.678254805" watchObservedRunningTime="2026-07-28 00:46:31.123358699 +0000 UTC m=+22.678861859"
Jul 28 00:46:31.170338 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.170296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qtd5t" podStartSLOduration=2.7928413340000002 podStartE2EDuration="22.170275396s" totalImagesPullingTime="19.377434062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:31.170002859 +0000 UTC m=+22.725506047" watchObservedRunningTime="2026-07-28 00:46:31.170275396 +0000 UTC m=+22.725778557"
Jul 28 00:46:31.184853 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:31.184810    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gf5ph" podStartSLOduration=2.902709018 podStartE2EDuration="22.184794529s" totalImagesPullingTime="19.282085511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:31.184601687 +0000 UTC m=+22.740104846" watchObservedRunningTime="2026-07-28 00:46:31.184794529 +0000 UTC m=+22.740297716"
Jul 28 00:46:32.081252 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.081208    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdnsv" event={"ID":"b423eba2-2ccd-469a-99a2-fca1cc47a898","Type":"ContainerStarted","Data":"4502a99c24385d7ed185abef95c354eb1d9665f1bf98bbd0dd5663f05a16ddd7"}
Jul 28 00:46:32.085586 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.085557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"ddf4e2098c4f40e65da879afa37c2bc334742bf2ff38874f7d1cc112d892e452"}
Jul 28 00:46:32.085733 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.085634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"23c6060bf8c5a3e7d81f6f9c2f1c4e4e49547ec9d8518df728e930a760e912ab"}
Jul 28 00:46:32.085733 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.085694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"da91b1ec50dd625a5ade61489fe7dbe6eefe3e81643f9af3c6962331d1f3636f"}
Jul 28 00:46:32.094715 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.094676    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pdnsv" podStartSLOduration=3.724852168 podStartE2EDuration="23.094661518s" totalImagesPullingTime="19.36980935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:32.094122351 +0000 UTC m=+23.649625545" watchObservedRunningTime="2026-07-28 00:46:32.094661518 +0000 UTC m=+23.650164679"
Jul 28 00:46:32.128115 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.128086    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 00:46:32.903159 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.902983    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T00:46:32.12810875Z","UUID":"24b9ba83-7895-4368-9659-ad9530ad1ea2","Handler":null,"Name":"","Endpoint":""}
Jul 28 00:46:32.906272 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.906215    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 00:46:32.906423 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.906285    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 00:46:32.939368 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.939337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:32.939542 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.939376    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:32.939542 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:32.939337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:32.939542 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:32.939477    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:32.939743 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:32.939572    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:32.939743 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:32.939651    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:33.089786 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:33.089750    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz" event={"ID":"c30d7d6a-eb19-4e83-aaf6-63edaccece49","Type":"ContainerStarted","Data":"55fa1c8fe25b690411032880c20b08f2ad3ad4e4edf51e250feffc1542e95892"}
Jul 28 00:46:34.094742 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:34.094512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz" event={"ID":"c30d7d6a-eb19-4e83-aaf6-63edaccece49","Type":"ContainerStarted","Data":"55e9fc11f713b053ebd8746c935e16c2821d464dc99841836e50ac0d8e9e2646"}
Jul 28 00:46:34.098207 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:34.098176    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"6f4be8c54c6da2b7356e988f5a528ec9eb5777db100106995e192b48b22b8403"}
Jul 28 00:46:34.112914 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:34.112866    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7m6bz" podStartSLOduration=3.13851907 podStartE2EDuration="25.112847957s" totalImagesPullingTime="21.974328887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:34.112175887 +0000 UTC m=+25.667679066" watchObservedRunningTime="2026-07-28 00:46:34.112847957 +0000 UTC m=+25.668351179"
Jul 28 00:46:34.939845 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:34.939808    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:34.940048 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:34.939814    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:34.940048 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:34.939926    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:34.940048 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:34.939954    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:34.940172 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:34.940053    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:34.940219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:34.940178    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:36.069833 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.069556    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:36.070550 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.070265    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:36.107081 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.107041    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" event={"ID":"a848cccd-6c9b-433f-8554-95b82faec0e6","Type":"ContainerStarted","Data":"f7baff56b0a981d50ea378ba09b456f3d8838e4da60bf955d0c1c1646f0956b1"}
Jul 28 00:46:36.107379 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.107358    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:36.109010 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.108989    2564 generic.go:356] "Generic (PLEG): container finished" podID="a339eb81-8e58-4560-8c0b-d37f7750813e" containerID="b474f2996ad8c3e8fefba43e8ae225cc4f5d91b6549e6b1e968dca0fe7281f93" exitCode=0
Jul 28 00:46:36.109134 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.109076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerDied","Data":"b474f2996ad8c3e8fefba43e8ae225cc4f5d91b6549e6b1e968dca0fe7281f93"}
Jul 28 00:46:36.125012 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.124985    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:36.132452 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.132401    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh" podStartSLOduration=7.564256782 podStartE2EDuration="27.132385683s" totalImagesPullingTime="19.568128901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:36.131253985 +0000 UTC m=+27.686757157" watchObservedRunningTime="2026-07-28 00:46:36.132385683 +0000 UTC m=+27.687888842"
Jul 28 00:46:36.939861 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.939827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:36.939861 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.939827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:36.940361 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:36.939972    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:36.940361 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:36.940027    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:36.940361 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:36.940121    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:36.940361 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:36.940190    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:37.112140 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.112113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:37.112621 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.112232    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:37.127202 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.127168    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:46:37.536984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.536950    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vvw7"]
Jul 28 00:46:37.537208 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.537064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:37.537208 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:37.537179    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:37.539785 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.539760    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z5mrk"]
Jul 28 00:46:37.539919 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.539853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:37.540041 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:37.539987    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:37.542749 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.542726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g88nz"]
Jul 28 00:46:37.542838 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:37.542812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:37.542913 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:37.542891    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:38.115739 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.115702    2564 generic.go:356] "Generic (PLEG): container finished" podID="a339eb81-8e58-4560-8c0b-d37f7750813e" containerID="b2c9383a9b1ec8f36968f58b85334dd242c55fe7db7b91430fdf76c6943b354d" exitCode=0
Jul 28 00:46:38.116228 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.115778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerDied","Data":"b2c9383a9b1ec8f36968f58b85334dd242c55fe7db7b91430fdf76c6943b354d"}
Jul 28 00:46:38.634542 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.634514    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:38.634711 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.634628    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 00:46:38.635102 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.635080    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qtd5t"
Jul 28 00:46:38.939921 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.939842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:38.940094 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:38.939981    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:38.940094 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:38.940036    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:38.940192 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:38.940138    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:39.939903 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:39.939732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:39.940308 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:39.940015    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:40.122362 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:40.122328    2564 generic.go:356] "Generic (PLEG): container finished" podID="a339eb81-8e58-4560-8c0b-d37f7750813e" containerID="f72b88b552c2a5dc0bdafa2e6cd50b1d9c1b7b901d1d8e2e0f2ed863cbb0c40f" exitCode=0
Jul 28 00:46:40.122511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:40.122397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerDied","Data":"f72b88b552c2a5dc0bdafa2e6cd50b1d9c1b7b901d1d8e2e0f2ed863cbb0c40f"}
Jul 28 00:46:40.939367 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:40.939335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:40.939583 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:40.939342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:40.939583 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:40.939478    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:40.939687 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:40.939578    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:41.939918 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:41.939878    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:41.940468 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:41.940037    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vvw7" podUID="8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8"
Jul 28 00:46:42.583997 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:42.583954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:42.584196 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:42.584036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:42.584196 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.584066    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:42.584196 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.584134    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:42.584196 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.584141    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret podName:8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:14.584123817 +0000 UTC m=+66.139626960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret") pod "global-pull-secret-syncer-6vvw7" (UID: "8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:42.584196 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.584182    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs podName:0da18fd8-305e-4996-a531-2f569665d052 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:14.584167328 +0000 UTC m=+66.139670478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs") pod "network-metrics-daemon-g88nz" (UID: "0da18fd8-305e-4996-a531-2f569665d052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:42.785073 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:42.785037    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:42.785263 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.785207    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:42.785263 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.785233    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:42.785263 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.785245    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bgcxk for pod openshift-network-diagnostics/network-check-target-z5mrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:42.785419 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.785312    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk podName:2640711a-54c5-435f-a218-e89af0639bcb nodeName:}" failed. No retries permitted until 2026-07-28 00:47:14.785291899 +0000 UTC m=+66.340795037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgcxk" (UniqueName: "kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk") pod "network-check-target-z5mrk" (UID: "2640711a-54c5-435f-a218-e89af0639bcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:42.939349 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:42.939272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:42.939482 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.939387    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g88nz" podUID="0da18fd8-305e-4996-a531-2f569665d052"
Jul 28 00:46:42.939553 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:42.939489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:42.939625 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:42.939602    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z5mrk" podUID="2640711a-54c5-435f-a218-e89af0639bcb"
Jul 28 00:46:43.299272 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.299190    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-111.ec2.internal" event="NodeReady"
Jul 28 00:46:43.299774 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.299358    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 00:46:43.330960 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.330880    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"]
Jul 28 00:46:43.332870 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.332847    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:43.334920 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.334890    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"]
Jul 28 00:46:43.336583 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.336403    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.336583 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.336457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.336766 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.336730    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 00:46:43.337195 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.336881    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v8mg6\""
Jul 28 00:46:43.337195 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.336965    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kkl92"]
Jul 28 00:46:43.337195 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.337150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.338836 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.338817    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"]
Jul 28 00:46:43.339015 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.338994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.340574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.340553    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"]
Jul 28 00:46:43.340688 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.340633    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"
Jul 28 00:46:43.341455 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341243    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 00:46:43.341455 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341258    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rnvgz\""
Jul 28 00:46:43.341455 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.341455 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 00:46:43.341455 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341411    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.341767 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341671    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mqlxp\""
Jul 28 00:46:43.341767 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341705    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 00:46:43.341767 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341717    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.341906 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341779    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.341906 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.341841    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 00:46:43.342166 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.342149    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-l64k4"]
Jul 28 00:46:43.342269 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.342255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.342639 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.342621    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.343929 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.343897    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.344038 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.343987    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d48b877f-xhtsx"]
Jul 28 00:46:43.344199 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.344184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.348841 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.347280    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.348841 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.347557    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.350133 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.350108    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.350300 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.350267    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2m7xw\""
Jul 28 00:46:43.350565 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.350547    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"]
Jul 28 00:46:43.352440 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.352123    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.352440 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.352184    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rbpvf\""
Jul 28 00:46:43.352440 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.350635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 00:46:43.352773 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.352523    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tzj5m\""
Jul 28 00:46:43.353447 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.353080    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 00:46:43.353447 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.353318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.353447 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.353419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.353631 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.353621    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69dbff6fc5-m2lf9"]
Jul 28 00:46:43.357681 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.350569    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 00:46:43.357681 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.354541    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 00:46:43.357681 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.355620    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 00:46:43.357681 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.355929    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qwcv6\""
Jul 28 00:46:43.359536 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.359515    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.359926 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.359902    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.360164 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360129    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.360318 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360297    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 00:46:43.360479 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360459    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 00:46:43.360624 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360362    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 00:46:43.360723 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 00:46:43.360787 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360761    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.360877 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360846    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 00:46:43.361009 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360928    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"]
Jul 28 00:46:43.361009 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.360994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"]
Jul 28 00:46:43.361206 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.361015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"]
Jul 28 00:46:43.361206 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.361037    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"]
Jul 28 00:46:43.362578 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.361038    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.362578 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.362432    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bvhj4\""
Jul 28 00:46:43.362705 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.362670    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 00:46:43.362761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.362730    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 00:46:43.363180 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.363158    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kkl92"]
Jul 28 00:46:43.363308 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.363198    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"]
Jul 28 00:46:43.363308 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.363300    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:43.366291 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.366274    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"]
Jul 28 00:46:43.366503 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.366465    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69dbff6fc5-m2lf9"]
Jul 28 00:46:43.366701 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.366661    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d48b877f-xhtsx"]
Jul 28 00:46:43.366896 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.366866    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7vx6j"]
Jul 28 00:46:43.367139 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 00:46:43.367304 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367286    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 00:46:43.367415 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367402    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qz9w8\""
Jul 28 00:46:43.368103 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.366366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"
Jul 28 00:46:43.368103 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h2f8w\""
Jul 28 00:46:43.368248 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367687    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 00:46:43.368248 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367723    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 00:46:43.368343 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.367816    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 00:46:43.369805 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.369782    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mrvdd"]
Jul 28 00:46:43.370333 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.370314    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.376603 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.376576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-54rrh\""
Jul 28 00:46:43.376922 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.376898    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-l64k4"]
Jul 28 00:46:43.379173 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.377156    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.379173 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.377458    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 00:46:43.379173 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.378118    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.379173 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.378359    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ktwkk\""
Jul 28 00:46:43.379173 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.378918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.380868 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.380843    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 00:46:43.381451 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.381287    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"]
Jul 28 00:46:43.383313 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.383291    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 00:46:43.383460 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.383439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"]
Jul 28 00:46:43.385357 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.385303    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7vx6j"]
Jul 28 00:46:43.386004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.385513    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 00:46:43.386004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.385529    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nzvbc\""
Jul 28 00:46:43.386004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.385775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 00:46:43.386004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.385822    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 00:46:43.387239 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.387216    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrvdd"]
Jul 28 00:46:43.388750 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.388730    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"]
Jul 28 00:46:43.490837 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.490805    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6dacd7-8d50-4764-8313-e15b794713c3-tmp\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.491072 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.490851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:43.491072 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.490895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntvr\" (UniqueName: \"kubernetes.io/projected/36669965-8a4a-4c7b-89c0-3b1b5e995c3a-kube-api-access-gntvr\") pod \"volume-data-source-validator-549c6d54cf-z8cgh\" (UID: \"36669965-8a4a-4c7b-89c0-3b1b5e995c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"
Jul 28 00:46:43.491072 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.490920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.491072 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491001    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29f05ced-501b-4e37-b673-706d1a80b7af-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:43.491072 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-ca-trust-extracted\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/05ad823c-b798-4bc0-95fe-2885ce46e53b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-image-registry-private-configuration\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491125    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zp9\" (UniqueName: \"kubernetes.io/projected/c9024cb5-e372-4d36-84f4-d9935078d587-kube-api-access-g8zp9\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e6dacd7-8d50-4764-8313-e15b794713c3-serving-cert\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ce6a7c-6678-4431-a4eb-6ceacba6faef-config-volume\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95x9r\" (UniqueName: \"kubernetes.io/projected/6e6dacd7-8d50-4764-8313-e15b794713c3-kube-api-access-95x9r\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491315    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rq7\" (UniqueName: \"kubernetes.io/projected/61ce6a7c-6678-4431-a4eb-6ceacba6faef-kube-api-access-w2rq7\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.491344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6wfp\" (UniqueName: \"kubernetes.io/projected/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-kube-api-access-c6wfp\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-serving-cert\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-installation-pull-secrets\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-bound-sa-token\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-certificates\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ce6a7c-6678-4431-a4eb-6ceacba6faef-tmp-dir\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5p8t\" (UniqueName: \"kubernetes.io/projected/493749e8-7d19-468f-a472-fceb6face574-kube-api-access-r5p8t\") pod \"ingress-canary-mrvdd\" (UID: \"493749e8-7d19-468f-a472-fceb6face574\") " pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sph\" (UniqueName: \"kubernetes.io/projected/4356f204-868b-4aa7-bfc2-bf271dbff21c-kube-api-access-26sph\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4356f204-868b-4aa7-bfc2-bf271dbff21c-config\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:43.491831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqb9l\" (UniqueName: \"kubernetes.io/projected/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-kube-api-access-wqb9l\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e6dacd7-8d50-4764-8313-e15b794713c3-service-ca-bundle\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.491997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqq5\" (UniqueName: \"kubernetes.io/projected/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-kube-api-access-5pqq5\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4q9q\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-kube-api-access-j4q9q\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492116    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-trusted-ca\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e6dacd7-8d50-4764-8313-e15b794713c3-snapshots\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e6dacd7-8d50-4764-8313-e15b794713c3-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.492276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/493749e8-7d19-468f-a472-fceb6face574-cert\") pod \"ingress-canary-mrvdd\" (UID: \"493749e8-7d19-468f-a472-fceb6face574\") " pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4356f204-868b-4aa7-bfc2-bf271dbff21c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-stats-auth\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-config\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bxt\" (UniqueName: \"kubernetes.io/projected/52a8208c-8481-4dcb-a1bd-942a0ca3a595-kube-api-access-h8bxt\") pod \"network-check-source-7d88cb55d8-hw728\" (UID: \"52a8208c-8481-4dcb-a1bd-942a0ca3a595\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77tp\" (UniqueName: \"kubernetes.io/projected/05ad823c-b798-4bc0-95fe-2885ce46e53b-kube-api-access-h77tp\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-default-certificate\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-trusted-ca\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.492672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.492577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.593312 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-image-registry-private-configuration\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.593312 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zp9\" (UniqueName: \"kubernetes.io/projected/c9024cb5-e372-4d36-84f4-d9935078d587-kube-api-access-g8zp9\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e6dacd7-8d50-4764-8313-e15b794713c3-serving-cert\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ce6a7c-6678-4431-a4eb-6ceacba6faef-config-volume\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95x9r\" (UniqueName: \"kubernetes.io/projected/6e6dacd7-8d50-4764-8313-e15b794713c3-kube-api-access-95x9r\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rq7\" (UniqueName: \"kubernetes.io/projected/61ce6a7c-6678-4431-a4eb-6ceacba6faef-kube-api-access-w2rq7\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6wfp\" (UniqueName: \"kubernetes.io/projected/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-kube-api-access-c6wfp\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-serving-cert\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-installation-pull-secrets\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.593539 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.593522    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 00:46:43.594044 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.593607    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls podName:61ce6a7c-6678-4431-a4eb-6ceacba6faef nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.093584566 +0000 UTC m=+35.649087722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls") pod "dns-default-7vx6j" (UID: "61ce6a7c-6678-4431-a4eb-6ceacba6faef") : secret "dns-default-metrics-tls" not found
Jul 28 00:46:43.594044 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-bound-sa-token\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.594044 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.593778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-certificates\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.594044 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.593913    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.093895707 +0000 UTC m=+35.649398859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : configmap references non-existent config key: service-ca.crt
Jul 28 00:46:43.594244 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ce6a7c-6678-4431-a4eb-6ceacba6faef-config-volume\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.594244 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ce6a7c-6678-4431-a4eb-6ceacba6faef-tmp-dir\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.594244 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594147    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5p8t\" (UniqueName: \"kubernetes.io/projected/493749e8-7d19-468f-a472-fceb6face574-kube-api-access-r5p8t\") pod \"ingress-canary-mrvdd\" (UID: \"493749e8-7d19-468f-a472-fceb6face574\") " pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.594244 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sph\" (UniqueName: \"kubernetes.io/projected/4356f204-868b-4aa7-bfc2-bf271dbff21c-kube-api-access-26sph\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.594244 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4356f204-868b-4aa7-bfc2-bf271dbff21c-config\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqb9l\" (UniqueName: \"kubernetes.io/projected/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-kube-api-access-wqb9l\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e6dacd7-8d50-4764-8313-e15b794713c3-service-ca-bundle\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqq5\" (UniqueName: \"kubernetes.io/projected/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-kube-api-access-5pqq5\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4q9q\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-kube-api-access-j4q9q\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-trusted-ca\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e6dacd7-8d50-4764-8313-e15b794713c3-snapshots\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.594488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e6dacd7-8d50-4764-8313-e15b794713c3-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/493749e8-7d19-468f-a472-fceb6face574-cert\") pod \"ingress-canary-mrvdd\" (UID: \"493749e8-7d19-468f-a472-fceb6face574\") " pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4356f204-868b-4aa7-bfc2-bf271dbff21c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-stats-auth\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-config\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bxt\" (UniqueName: \"kubernetes.io/projected/52a8208c-8481-4dcb-a1bd-942a0ca3a595-kube-api-access-h8bxt\") pod \"network-check-source-7d88cb55d8-hw728\" (UID: \"52a8208c-8481-4dcb-a1bd-942a0ca3a595\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77tp\" (UniqueName: \"kubernetes.io/projected/05ad823c-b798-4bc0-95fe-2885ce46e53b-kube-api-access-h77tp\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-default-certificate\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-trusted-ca\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ce6a7c-6678-4431-a4eb-6ceacba6faef-tmp-dir\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4356f204-868b-4aa7-bfc2-bf271dbff21c-config\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.594950 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.594924    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.594994    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls podName:c9024cb5-e372-4d36-84f4-d9935078d587 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.094980893 +0000 UTC m=+35.650484030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls") pod "cluster-samples-operator-6748489656-xj2zm" (UID: "c9024cb5-e372-4d36-84f4-d9935078d587") : secret "samples-operator-tls" not found
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6dacd7-8d50-4764-8313-e15b794713c3-tmp\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntvr\" (UniqueName: \"kubernetes.io/projected/36669965-8a4a-4c7b-89c0-3b1b5e995c3a-kube-api-access-gntvr\") pod \"volume-data-source-validator-549c6d54cf-z8cgh\" (UID: \"36669965-8a4a-4c7b-89c0-3b1b5e995c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.595116    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.595188    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.095174099 +0000 UTC m=+35.650677250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : secret "router-metrics-certs-default" not found
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29f05ced-501b-4e37-b673-706d1a80b7af-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595231    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-ca-trust-extracted\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.595549 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/05ad823c-b798-4bc0-95fe-2885ce46e53b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.596086 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.596032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/05ad823c-b798-4bc0-95fe-2885ce46e53b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.596489 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.595136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6dacd7-8d50-4764-8313-e15b794713c3-tmp\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.597113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-trusted-ca\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.597337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-trusted-ca\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.597436    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.597495    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls podName:05ad823c-b798-4bc0-95fe-2885ce46e53b nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.097479004 +0000 UTC m=+35.652982151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9jrqd" (UID: "05ad823c-b798-4bc0-95fe-2885ce46e53b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.597610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e6dacd7-8d50-4764-8313-e15b794713c3-service-ca-bundle\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.594375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-certificates\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.598783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-ca-trust-extracted\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.599138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-serving-cert\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.599232    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.599233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-default-certificate\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.599280    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert podName:29f05ced-501b-4e37-b673-706d1a80b7af nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.099264031 +0000 UTC m=+35.654767171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v5ct8" (UID: "29f05ced-501b-4e37-b673-706d1a80b7af") : secret "networking-console-plugin-cert" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.599528    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.599543    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbff6fc5-m2lf9: secret "image-registry-tls" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:43.599580    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls podName:f0660dbe-4d29-4a5c-84b2-6e25b6af7603 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:44.099567284 +0000 UTC m=+35.655070439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls") pod "image-registry-69dbff6fc5-m2lf9" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603") : secret "image-registry-tls" not found
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.599614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-config\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.600634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.599681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e6dacd7-8d50-4764-8313-e15b794713c3-snapshots\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.599916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-image-registry-private-configuration\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.600163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-installation-pull-secrets\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.600282    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.600413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e6dacd7-8d50-4764-8313-e15b794713c3-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.600709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e6dacd7-8d50-4764-8313-e15b794713c3-serving-cert\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.601062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-stats-auth\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.601315    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.601394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29f05ced-501b-4e37-b673-706d1a80b7af-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:43.601538 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.601458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4356f204-868b-4aa7-bfc2-bf271dbff21c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.604109 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.603968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95x9r\" (UniqueName: \"kubernetes.io/projected/6e6dacd7-8d50-4764-8313-e15b794713c3-kube-api-access-95x9r\") pod \"insights-operator-5885fdb759-kkl92\" (UID: \"6e6dacd7-8d50-4764-8313-e15b794713c3\") " pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.606308 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.606277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6wfp\" (UniqueName: \"kubernetes.io/projected/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-kube-api-access-c6wfp\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:43.607698 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.607669    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sph\" (UniqueName: \"kubernetes.io/projected/4356f204-868b-4aa7-bfc2-bf271dbff21c-kube-api-access-26sph\") pod \"service-ca-operator-5d6c75f84f-65gc9\" (UID: \"4356f204-868b-4aa7-bfc2-bf271dbff21c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.610907 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.610482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqq5\" (UniqueName: \"kubernetes.io/projected/91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4-kube-api-access-5pqq5\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc692\" (UID: \"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.610907 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.610832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4q9q\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-kube-api-access-j4q9q\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.616232 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.611426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bxt\" (UniqueName: \"kubernetes.io/projected/52a8208c-8481-4dcb-a1bd-942a0ca3a595-kube-api-access-h8bxt\") pod \"network-check-source-7d88cb55d8-hw728\" (UID: \"52a8208c-8481-4dcb-a1bd-942a0ca3a595\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"
Jul 28 00:46:43.619676 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.619645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5p8t\" (UniqueName: \"kubernetes.io/projected/493749e8-7d19-468f-a472-fceb6face574-kube-api-access-r5p8t\") pod \"ingress-canary-mrvdd\" (UID: \"493749e8-7d19-468f-a472-fceb6face574\") " pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.619757 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.619645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zp9\" (UniqueName: \"kubernetes.io/projected/c9024cb5-e372-4d36-84f4-d9935078d587-kube-api-access-g8zp9\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:43.619922 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.619904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/493749e8-7d19-468f-a472-fceb6face574-cert\") pod \"ingress-canary-mrvdd\" (UID: \"493749e8-7d19-468f-a472-fceb6face574\") " pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.621983 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.621958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntvr\" (UniqueName: \"kubernetes.io/projected/36669965-8a4a-4c7b-89c0-3b1b5e995c3a-kube-api-access-gntvr\") pod \"volume-data-source-validator-549c6d54cf-z8cgh\" (UID: \"36669965-8a4a-4c7b-89c0-3b1b5e995c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"
Jul 28 00:46:43.622117 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.622021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqb9l\" (UniqueName: \"kubernetes.io/projected/b707cfdd-c2b5-4dc8-b386-b096fb2335c1-kube-api-access-wqb9l\") pod \"console-operator-584dcb6b54-l64k4\" (UID: \"b707cfdd-c2b5-4dc8-b386-b096fb2335c1\") " pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.622608 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.622588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rq7\" (UniqueName: \"kubernetes.io/projected/61ce6a7c-6678-4431-a4eb-6ceacba6faef-kube-api-access-w2rq7\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:43.622722 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.622704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-bound-sa-token\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:43.623147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.623127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77tp\" (UniqueName: \"kubernetes.io/projected/05ad823c-b798-4bc0-95fe-2885ce46e53b-kube-api-access-h77tp\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:43.673184 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.673145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-kkl92"
Jul 28 00:46:43.697732 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.697699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"
Jul 28 00:46:43.698808 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.698766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"
Jul 28 00:46:43.705833 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.705807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:43.719773 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.719742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"
Jul 28 00:46:43.747000 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.746391    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"
Jul 28 00:46:43.751635 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.750203    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrvdd"
Jul 28 00:46:43.918271 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.917858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kkl92"]
Jul 28 00:46:43.939661 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.939029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:46:43.942484 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.941990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 00:46:43.947634 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.947553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692"]
Jul 28 00:46:43.954638 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.953706    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh"]
Jul 28 00:46:43.959568 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:43.959428    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a9d9d1_d089_4908_99bb_c0e7ea3e1ea4.slice/crio-c72e5d915750dc16e014fa0277f00ca6d482c13235f26f5e9c9a46e33db97f1d WatchSource:0}: Error finding container c72e5d915750dc16e014fa0277f00ca6d482c13235f26f5e9c9a46e33db97f1d: Status 404 returned error can't find the container with id c72e5d915750dc16e014fa0277f00ca6d482c13235f26f5e9c9a46e33db97f1d
Jul 28 00:46:43.963450 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:43.963418    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36669965_8a4a_4c7b_89c0_3b1b5e995c3a.slice/crio-7e6b519c2271d284fffa29f2d762f925016b12d1bc50166a99ea1872bd7a6d94 WatchSource:0}: Error finding container 7e6b519c2271d284fffa29f2d762f925016b12d1bc50166a99ea1872bd7a6d94: Status 404 returned error can't find the container with id 7e6b519c2271d284fffa29f2d762f925016b12d1bc50166a99ea1872bd7a6d94
Jul 28 00:46:43.974920 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.974850    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-l64k4"]
Jul 28 00:46:43.979245 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:43.979087    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9"]
Jul 28 00:46:43.979496 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:43.979458    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb707cfdd_c2b5_4dc8_b386_b096fb2335c1.slice/crio-e48947464e7d18e8ad2e2da772fa758fe75b01abd88045785d1e6672d79e4787 WatchSource:0}: Error finding container e48947464e7d18e8ad2e2da772fa758fe75b01abd88045785d1e6672d79e4787: Status 404 returned error can't find the container with id e48947464e7d18e8ad2e2da772fa758fe75b01abd88045785d1e6672d79e4787
Jul 28 00:46:43.981697 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:43.981247    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4356f204_868b_4aa7_bfc2_bf271dbff21c.slice/crio-86212a616954cfb67c2dc4bc98b622aa8e8fd5a57972faf3fe1361880aeaf868 WatchSource:0}: Error finding container 86212a616954cfb67c2dc4bc98b622aa8e8fd5a57972faf3fe1361880aeaf868: Status 404 returned error can't find the container with id 86212a616954cfb67c2dc4bc98b622aa8e8fd5a57972faf3fe1361880aeaf868
Jul 28 00:46:44.000962 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.000871    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrvdd"]
Jul 28 00:46:44.003590 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:44.003554    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493749e8_7d19_468f_a472_fceb6face574.slice/crio-ebbd51535807fcd5319110e1cc6bb8ba225da9f4a70e01cadcf5204121d18381 WatchSource:0}: Error finding container ebbd51535807fcd5319110e1cc6bb8ba225da9f4a70e01cadcf5204121d18381: Status 404 returned error can't find the container with id ebbd51535807fcd5319110e1cc6bb8ba225da9f4a70e01cadcf5204121d18381
Jul 28 00:46:44.033461 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.033428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728"]
Jul 28 00:46:44.036773 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:44.036739    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a8208c_8481_4dcb_a1bd_942a0ca3a595.slice/crio-c779abcff500500df352e2c8d0ac2d22701e29bd52e3115390f15a2ff4b7f581 WatchSource:0}: Error finding container c779abcff500500df352e2c8d0ac2d22701e29bd52e3115390f15a2ff4b7f581: Status 404 returned error can't find the container with id c779abcff500500df352e2c8d0ac2d22701e29bd52e3115390f15a2ff4b7f581
Jul 28 00:46:44.105231 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:44.105370 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105308    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 00:46:44.105405 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105377    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls podName:61ce6a7c-6678-4431-a4eb-6ceacba6faef nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.105354666 +0000 UTC m=+36.660857804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls") pod "dns-default-7vx6j" (UID: "61ce6a7c-6678-4431-a4eb-6ceacba6faef") : secret "dns-default-metrics-tls" not found
Jul 28 00:46:44.105405 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:44.105521 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:44.105521 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:44.105610 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:44.105610 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105566    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.105552204 +0000 UTC m=+36.661055346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : configmap references non-existent config key: service-ca.crt
Jul 28 00:46:44.105610 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:44.105753 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105614    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 00:46:44.105753 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105625    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbff6fc5-m2lf9: secret "image-registry-tls" not found
Jul 28 00:46:44.105753 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.105653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:44.105753 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105664    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls podName:f0660dbe-4d29-4a5c-84b2-6e25b6af7603 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.105653938 +0000 UTC m=+36.661157075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls") pod "image-registry-69dbff6fc5-m2lf9" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603") : secret "image-registry-tls" not found
Jul 28 00:46:44.105753 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105737    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105771    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls podName:05ad823c-b798-4bc0-95fe-2885ce46e53b nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.105760529 +0000 UTC m=+36.661263685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9jrqd" (UID: "05ad823c-b798-4bc0-95fe-2885ce46e53b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105773    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105805    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert podName:29f05ced-501b-4e37-b673-706d1a80b7af nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.105795081 +0000 UTC m=+36.661298222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v5ct8" (UID: "29f05ced-501b-4e37-b673-706d1a80b7af") : secret "networking-console-plugin-cert" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105850    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105860    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105889    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls podName:c9024cb5-e372-4d36-84f4-d9935078d587 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.10587844 +0000 UTC m=+36.661381591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls") pod "cluster-samples-operator-6748489656-xj2zm" (UID: "c9024cb5-e372-4d36-84f4-d9935078d587") : secret "samples-operator-tls" not found
Jul 28 00:46:44.105922 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:44.105907    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.105901214 +0000 UTC m=+36.661404352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : secret "router-metrics-certs-default" not found
Jul 28 00:46:44.133845 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.133765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrvdd" event={"ID":"493749e8-7d19-468f-a472-fceb6face574","Type":"ContainerStarted","Data":"ebbd51535807fcd5319110e1cc6bb8ba225da9f4a70e01cadcf5204121d18381"}
Jul 28 00:46:44.135181 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.135147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728" event={"ID":"52a8208c-8481-4dcb-a1bd-942a0ca3a595","Type":"ContainerStarted","Data":"c779abcff500500df352e2c8d0ac2d22701e29bd52e3115390f15a2ff4b7f581"}
Jul 28 00:46:44.136607 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.136569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh" event={"ID":"36669965-8a4a-4c7b-89c0-3b1b5e995c3a","Type":"ContainerStarted","Data":"7e6b519c2271d284fffa29f2d762f925016b12d1bc50166a99ea1872bd7a6d94"}
Jul 28 00:46:44.137885 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.137852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9" event={"ID":"4356f204-868b-4aa7-bfc2-bf271dbff21c","Type":"ContainerStarted","Data":"86212a616954cfb67c2dc4bc98b622aa8e8fd5a57972faf3fe1361880aeaf868"}
Jul 28 00:46:44.139157 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.139134    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692" event={"ID":"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4","Type":"ContainerStarted","Data":"c72e5d915750dc16e014fa0277f00ca6d482c13235f26f5e9c9a46e33db97f1d"}
Jul 28 00:46:44.140282 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.140262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" event={"ID":"b707cfdd-c2b5-4dc8-b386-b096fb2335c1","Type":"ContainerStarted","Data":"e48947464e7d18e8ad2e2da772fa758fe75b01abd88045785d1e6672d79e4787"}
Jul 28 00:46:44.141471 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.141449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kkl92" event={"ID":"6e6dacd7-8d50-4764-8313-e15b794713c3","Type":"ContainerStarted","Data":"75b6153fca718b514bb22efb337432dd7e8e58ee39b38733547b4c19aeb7095b"}
Jul 28 00:46:44.939695 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.939661    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:46:44.940409 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.939671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:46:44.942279 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.942257    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 00:46:44.943125 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.943109    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kdcl9\""
Jul 28 00:46:44.943256 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:44.943112    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkfth\""
Jul 28 00:46:45.115797 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.115760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:45.115797 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.115802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.115846    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.115873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.115913    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.115952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116016    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116024    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls podName:61ce6a7c-6678-4431-a4eb-6ceacba6faef nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116004844 +0000 UTC m=+38.671507983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls") pod "dns-default-7vx6j" (UID: "61ce6a7c-6678-4431-a4eb-6ceacba6faef") : secret "dns-default-metrics-tls" not found
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116046    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116056    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls podName:c9024cb5-e372-4d36-84f4-d9935078d587 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116046158 +0000 UTC m=+38.671549300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls") pod "cluster-samples-operator-6748489656-xj2zm" (UID: "c9024cb5-e372-4d36-84f4-d9935078d587") : secret "samples-operator-tls" not found
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116059    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbff6fc5-m2lf9: secret "image-registry-tls" not found
Jul 28 00:46:45.116064 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116072    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116064845 +0000 UTC m=+38.671567988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : configmap references non-existent config key: service-ca.crt
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.116095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116104    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls podName:f0660dbe-4d29-4a5c-84b2-6e25b6af7603 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116088264 +0000 UTC m=+38.671591419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls") pod "image-registry-69dbff6fc5-m2lf9" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603") : secret "image-registry-tls" not found
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116134    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116155    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.116162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116181    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116166327 +0000 UTC m=+38.671669466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : secret "router-metrics-certs-default" not found
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116215    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls podName:05ad823c-b798-4bc0-95fe-2885ce46e53b nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116205065 +0000 UTC m=+38.671708252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9jrqd" (UID: "05ad823c-b798-4bc0-95fe-2885ce46e53b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116265    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 00:46:45.116488 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:45.116308    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert podName:29f05ced-501b-4e37-b673-706d1a80b7af nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.116294615 +0000 UTC m=+38.671797767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v5ct8" (UID: "29f05ced-501b-4e37-b673-706d1a80b7af") : secret "networking-console-plugin-cert" not found
Jul 28 00:46:45.274097 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.273988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:45.375744 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.375694    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:45.457574 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.457532    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:45.522768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.522721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:45.619640 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:45.619612    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:46.164222 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:46.164115    2564 generic.go:356] "Generic (PLEG): container finished" podID="a339eb81-8e58-4560-8c0b-d37f7750813e" containerID="025be92c271a47e920bff72c199afaede4b23fadefe8952a6467a8dff8f9d7b9" exitCode=0
Jul 28 00:46:46.164222 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:46.164182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerDied","Data":"025be92c271a47e920bff72c199afaede4b23fadefe8952a6467a8dff8f9d7b9"}
Jul 28 00:46:46.214963 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:46.214809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:46.275422 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:46.275373    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:46.652814 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:46.652473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:46.711003 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:46.710775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.139296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.139433    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.139497    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.139477372 +0000 UTC m=+42.694980561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : secret "router-metrics-certs-default" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.139873    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.139888    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbff6fc5-m2lf9: secret "image-registry-tls" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.139948    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls podName:f0660dbe-4d29-4a5c-84b2-6e25b6af7603 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.139918736 +0000 UTC m=+42.695421876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls") pod "image-registry-69dbff6fc5-m2lf9" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603") : secret "image-registry-tls" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140007    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140041    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls podName:05ad823c-b798-4bc0-95fe-2885ce46e53b nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.140030093 +0000 UTC m=+42.695533238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9jrqd" (UID: "05ad823c-b798-4bc0-95fe-2885ce46e53b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140090    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 00:46:47.140534 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140120    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert podName:29f05ced-501b-4e37-b673-706d1a80b7af nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.140109678 +0000 UTC m=+42.695612824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v5ct8" (UID: "29f05ced-501b-4e37-b673-706d1a80b7af") : secret "networking-console-plugin-cert" not found
Jul 28 00:46:47.141462 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140204    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 00:46:47.141462 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140322    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls podName:61ce6a7c-6678-4431-a4eb-6ceacba6faef nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.140274812 +0000 UTC m=+42.695777952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls") pod "dns-default-7vx6j" (UID: "61ce6a7c-6678-4431-a4eb-6ceacba6faef") : secret "dns-default-metrics-tls" not found
Jul 28 00:46:47.141462 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140418    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.140408982 +0000 UTC m=+42.695912120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : configmap references non-existent config key: service-ca.crt
Jul 28 00:46:47.141462 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140474    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 00:46:47.141462 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:47.140503    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls podName:c9024cb5-e372-4d36-84f4-d9935078d587 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:51.140493838 +0000 UTC m=+42.695996981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls") pod "cluster-samples-operator-6748489656-xj2zm" (UID: "c9024cb5-e372-4d36-84f4-d9935078d587") : secret "samples-operator-tls" not found
Jul 28 00:46:47.178609 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.178572    2564 generic.go:356] "Generic (PLEG): container finished" podID="a339eb81-8e58-4560-8c0b-d37f7750813e" containerID="cd742b462b998e3d52c7eec525c0929db403589a5a691bf99da9ab712447fb7d" exitCode=0
Jul 28 00:46:47.179989 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:47.179238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerDied","Data":"cd742b462b998e3d52c7eec525c0929db403589a5a691bf99da9ab712447fb7d"}
Jul 28 00:46:48.185898 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.185526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh" event={"ID":"36669965-8a4a-4c7b-89c0-3b1b5e995c3a","Type":"ContainerStarted","Data":"b86af4e60f81902415df567abf9c0e34386ffc3e11a685cf78a720cba37827ce"}
Jul 28 00:46:48.192959 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.191093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" event={"ID":"a339eb81-8e58-4560-8c0b-d37f7750813e","Type":"ContainerStarted","Data":"a1b3124d01d383f05f14bfc981cbb4651d280a5ae1c44133bc81bf9617dc4ed8"}
Jul 28 00:46:48.200361 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.200305    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-z8cgh" podStartSLOduration=35.809012821 podStartE2EDuration="39.200285723s" totalImagesPullingTime="3.391272902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:48.199859453 +0000 UTC m=+39.755362637" watchObservedRunningTime="2026-07-28 00:46:48.200285723 +0000 UTC m=+39.755788862"
Jul 28 00:46:48.220516 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.220431    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jrzvt" podStartSLOduration=7.776184462 podStartE2EDuration="39.220409568s" totalImagesPullingTime="31.444225106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:48.219452547 +0000 UTC m=+39.774955706" watchObservedRunningTime="2026-07-28 00:46:48.220409568 +0000 UTC m=+39.775912728"
Jul 28 00:46:48.343606 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.343570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:48.400413 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.400321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:48.670247 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.670161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:48.731561 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:48.731519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:50.392129 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:50.392094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:50.495497 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:50.495432    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:51.178589 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:51.178589 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:51.178733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178735    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178766    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178796    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178818    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls podName:05ad823c-b798-4bc0-95fe-2885ce46e53b nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178795255 +0000 UTC m=+50.734298397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9jrqd" (UID: "05ad823c-b798-4bc0-95fe-2885ce46e53b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:51.178831 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178836    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert podName:29f05ced-501b-4e37-b673-706d1a80b7af nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178829345 +0000 UTC m=+50.734332482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v5ct8" (UID: "29f05ced-501b-4e37-b673-706d1a80b7af") : secret "networking-console-plugin-cert" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178799    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178846    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178840778 +0000 UTC m=+50.734343916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : configmap references non-existent config key: service-ca.crt
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178739    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178872    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbff6fc5-m2lf9: secret "image-registry-tls" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178847    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178860    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls podName:61ce6a7c-6678-4431-a4eb-6ceacba6faef nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178854435 +0000 UTC m=+50.734357573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls") pod "dns-default-7vx6j" (UID: "61ce6a7c-6678-4431-a4eb-6ceacba6faef") : secret "dns-default-metrics-tls" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178916    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls podName:c9024cb5-e372-4d36-84f4-d9935078d587 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178908106 +0000 UTC m=+50.734411245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls") pod "cluster-samples-operator-6748489656-xj2zm" (UID: "c9024cb5-e372-4d36-84f4-d9935078d587") : secret "samples-operator-tls" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178926    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178921602 +0000 UTC m=+50.734424740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : secret "router-metrics-certs-default" not found
Jul 28 00:46:51.179219 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:51.178952    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls podName:f0660dbe-4d29-4a5c-84b2-6e25b6af7603 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:59.178930915 +0000 UTC m=+50.734434053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls") pod "image-registry-69dbff6fc5-m2lf9" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603") : secret "image-registry-tls" not found
Jul 28 00:46:53.208222 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.207333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrvdd" event={"ID":"493749e8-7d19-468f-a472-fceb6face574","Type":"ContainerStarted","Data":"25be1769e0067b16484be096954ccabf2287908a3fcb08cd443dc1af2a171633"}
Jul 28 00:46:53.210682 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.210226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728" event={"ID":"52a8208c-8481-4dcb-a1bd-942a0ca3a595","Type":"ContainerStarted","Data":"43f1731f53bd211308d79f574790d46f16bcf47ffb98a42d948ba0fee8b8a506"}
Jul 28 00:46:53.213326 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.213249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9" event={"ID":"4356f204-868b-4aa7-bfc2-bf271dbff21c","Type":"ContainerStarted","Data":"dadc2cdeba17a6862c1141937d3b4a3fd933d46d3eee1e5769924c4dcf6fd501"}
Jul 28 00:46:53.215338 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.215307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692" event={"ID":"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4","Type":"ContainerStarted","Data":"3c16e8b90e0a6b733df1eabe46cfcb132f3854d2f07fef64ee38509d902e9cd6"}
Jul 28 00:46:53.217466 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.217390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" event={"ID":"b707cfdd-c2b5-4dc8-b386-b096fb2335c1","Type":"ContainerStarted","Data":"1af6e45e02c287a3ba66f4e37deee3536b340408b796802d095e1aef12871f82"}
Jul 28 00:46:53.217964 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.217887    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:46:53.219363 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.219339    2564 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-l64k4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Jul 28 00:46:53.219527 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.219499    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Jul 28 00:46:53.220443 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.220015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kkl92" event={"ID":"6e6dacd7-8d50-4764-8313-e15b794713c3","Type":"ContainerStarted","Data":"5546af409280c0e7ef1eef5ea214c9b4087f27864dde2841d109792677a701b5"}
Jul 28 00:46:53.223660 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.223616    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mrvdd" podStartSLOduration=1.268502305 podStartE2EDuration="10.223586118s" totalImagesPullingTime="8.955083813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:53.222787775 +0000 UTC m=+44.778290952" watchObservedRunningTime="2026-07-28 00:46:53.223586118 +0000 UTC m=+44.779089280"
Jul 28 00:46:53.269236 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.262426    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-kkl92" podStartSLOduration=35.236781913 podStartE2EDuration="44.262405796s" totalImagesPullingTime="9.025623883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:53.261285997 +0000 UTC m=+44.816789159" watchObservedRunningTime="2026-07-28 00:46:53.262405796 +0000 UTC m=+44.817908957"
Jul 28 00:46:53.312024 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.311963    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692" podStartSLOduration=34.313195715 podStartE2EDuration="43.31192476s" totalImagesPullingTime="8.998729045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:53.292611445 +0000 UTC m=+44.848114599" watchObservedRunningTime="2026-07-28 00:46:53.31192476 +0000 UTC m=+44.867427921"
Jul 28 00:46:53.312841 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.312768    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9" podStartSLOduration=32.337503944 podStartE2EDuration="41.312756233s" totalImagesPullingTime="8.975252289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:53.312387526 +0000 UTC m=+44.867890686" watchObservedRunningTime="2026-07-28 00:46:53.312756233 +0000 UTC m=+44.868259394"
Jul 28 00:46:53.347040 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.346974    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hw728" podStartSLOduration=31.415330542 podStartE2EDuration="40.346928442s" totalImagesPullingTime="8.9315979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:53.346871165 +0000 UTC m=+44.902374351" watchObservedRunningTime="2026-07-28 00:46:53.346928442 +0000 UTC m=+44.902431602"
Jul 28 00:46:53.347462 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.347431    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podStartSLOduration=35.368082605 podStartE2EDuration="44.347421939s" totalImagesPullingTime="8.979339334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:53.328499287 +0000 UTC m=+44.884002448" watchObservedRunningTime="2026-07-28 00:46:53.347421939 +0000 UTC m=+44.902925099"
Jul 28 00:46:53.707088 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.707043    2564 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-l64k4 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Jul 28 00:46:53.707291 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:53.707120    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Jul 28 00:46:54.135077 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.135039    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.193147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.193114    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q5hrw"]
Jul 28 00:46:54.215878 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.215848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.224763 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.224741    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q5hrw"]
Jul 28 00:46:54.224926 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.224844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.225576 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.225551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/0.log"
Jul 28 00:46:54.225699 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.225600    2564 generic.go:356] "Generic (PLEG): container finished" podID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1" containerID="1af6e45e02c287a3ba66f4e37deee3536b340408b796802d095e1aef12871f82" exitCode=255
Jul 28 00:46:54.225758 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.225675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" event={"ID":"b707cfdd-c2b5-4dc8-b386-b096fb2335c1","Type":"ContainerDied","Data":"1af6e45e02c287a3ba66f4e37deee3536b340408b796802d095e1aef12871f82"}
Jul 28 00:46:54.226004 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.225983    2564 scope.go:117] "RemoveContainer" containerID="1af6e45e02c287a3ba66f4e37deee3536b340408b796802d095e1aef12871f82"
Jul 28 00:46:54.227443 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.227416    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 00:46:54.227569 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.227416    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 00:46:54.228330 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.228297    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f27qr\""
Jul 28 00:46:54.274922 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.274895    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.307304 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.307272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qq6\" (UniqueName: \"kubernetes.io/projected/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-api-access-j7qq6\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.307543 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.307511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.307619 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.307579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bcdb825-e508-4e01-ad7b-65915bbd8b91-crio-socket\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.307619 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.307606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.307872 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.307855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bcdb825-e508-4e01-ad7b-65915bbd8b91-data-volume\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.333752 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.333721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.408977 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.408952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bcdb825-e508-4e01-ad7b-65915bbd8b91-data-volume\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409071 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.409015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qq6\" (UniqueName: \"kubernetes.io/projected/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-api-access-j7qq6\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409071 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.409043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409165 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.409076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bcdb825-e508-4e01-ad7b-65915bbd8b91-crio-socket\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409216 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.409188    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:54.409216 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.409199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409303 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.409219    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bcdb825-e508-4e01-ad7b-65915bbd8b91-crio-socket\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409303 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.409271    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:54.909249188 +0000 UTC m=+46.464752341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:54.409303 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.409266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bcdb825-e508-4e01-ad7b-65915bbd8b91-data-volume\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.409406 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.409310    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 00:46:54.409406 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.409372    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:54.909356265 +0000 UTC m=+46.464859415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : configmap "kube-rbac-proxy" not found
Jul 28 00:46:54.421864 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.421841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qq6\" (UniqueName: \"kubernetes.io/projected/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-api-access-j7qq6\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.434041 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.434004    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.776344 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.776305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.791395 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.791361    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"]
Jul 28 00:46:54.815146 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.815117    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"]
Jul 28 00:46:54.815299 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.815240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"
Jul 28 00:46:54.818605 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.818582    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 00:46:54.819169 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.818994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dtklk\""
Jul 28 00:46:54.819169 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.819043    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:54.913465 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.913409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:54.914352 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.914321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.914469 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.914373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:54.914469 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:54.914440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvthk\" (UniqueName: \"kubernetes.io/projected/a1b37d41-216e-418d-a163-e238c70acb9e-kube-api-access-pvthk\") pod \"migrator-6865c9967-ggxp4\" (UID: \"a1b37d41-216e-418d-a163-e238c70acb9e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"
Jul 28 00:46:54.914469 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.914449    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:54.914568 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.914505    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:55.914490249 +0000 UTC m=+47.469993390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:54.914568 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.914505    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 00:46:54.914632 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:54.914580    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:55.914570704 +0000 UTC m=+47.470073846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : configmap "kube-rbac-proxy" not found
Jul 28 00:46:55.013671 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.013627    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:55.015715 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.015683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvthk\" (UniqueName: \"kubernetes.io/projected/a1b37d41-216e-418d-a163-e238c70acb9e-kube-api-access-pvthk\") pod \"migrator-6865c9967-ggxp4\" (UID: \"a1b37d41-216e-418d-a163-e238c70acb9e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"
Jul 28 00:46:55.023562 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.023543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvthk\" (UniqueName: \"kubernetes.io/projected/a1b37d41-216e-418d-a163-e238c70acb9e-kube-api-access-pvthk\") pod \"migrator-6865c9967-ggxp4\" (UID: \"a1b37d41-216e-418d-a163-e238c70acb9e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"
Jul 28 00:46:55.124971 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.124870    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"
Jul 28 00:46:55.188637 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.188611    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66whq_cfef6df0-7787-4f83-8768-06920413b08a/dns-node-resolver/0.log"
Jul 28 00:46:55.230809 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.230786    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:46:55.231290 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.231271    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/0.log"
Jul 28 00:46:55.231390 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.231316    2564 generic.go:356] "Generic (PLEG): container finished" podID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1" containerID="7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785" exitCode=255
Jul 28 00:46:55.231390 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.231364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" event={"ID":"b707cfdd-c2b5-4dc8-b386-b096fb2335c1","Type":"ContainerDied","Data":"7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785"}
Jul 28 00:46:55.231529 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.231425    2564 scope.go:117] "RemoveContainer" containerID="1af6e45e02c287a3ba66f4e37deee3536b340408b796802d095e1aef12871f82"
Jul 28 00:46:55.231694 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.231676    2564 scope.go:117] "RemoveContainer" containerID="7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785"
Jul 28 00:46:55.231912 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:55.231882    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-l64k4_openshift-console-operator(b707cfdd-c2b5-4dc8-b386-b096fb2335c1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1"
Jul 28 00:46:55.268044 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.268013    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4"]
Jul 28 00:46:55.270772 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:55.270746    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b37d41_216e_418d_a163_e238c70acb9e.slice/crio-0f6f143d7db839a461fb2b4b7c8ef7acf2af52c8fc317bd772970e3422779225 WatchSource:0}: Error finding container 0f6f143d7db839a461fb2b4b7c8ef7acf2af52c8fc317bd772970e3422779225: Status 404 returned error can't find the container with id 0f6f143d7db839a461fb2b4b7c8ef7acf2af52c8fc317bd772970e3422779225
Jul 28 00:46:55.924825 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.924785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:55.925065 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:55.924859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:55.925065 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:55.924964    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:55.925065 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:55.925020    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 00:46:55.925065 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:55.925043    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:57.9250211 +0000 UTC m=+49.480524241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:55.925291 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:55.925073    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:57.9250568 +0000 UTC m=+49.480559938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : configmap "kube-rbac-proxy" not found
Jul 28 00:46:56.237628 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.237598    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:46:56.238135 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.238027    2564 scope.go:117] "RemoveContainer" containerID="7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785"
Jul 28 00:46:56.238241 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:56.238219    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-l64k4_openshift-console-operator(b707cfdd-c2b5-4dc8-b386-b096fb2335c1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1"
Jul 28 00:46:56.239089 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.239066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4" event={"ID":"a1b37d41-216e-418d-a163-e238c70acb9e","Type":"ContainerStarted","Data":"0f6f143d7db839a461fb2b4b7c8ef7acf2af52c8fc317bd772970e3422779225"}
Jul 28 00:46:56.439273 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.439238    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-cv8g9"]
Jul 28 00:46:56.452889 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.452859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-cv8g9"]
Jul 28 00:46:56.453048 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.452977    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.455543 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.455389    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cm6gl\""
Jul 28 00:46:56.455543 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.455408    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 00:46:56.455543 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.455394    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 00:46:56.455543 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.455461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 00:46:56.455543 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.455480    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 00:46:56.472392 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.472367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:56.531276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.531197    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5qq\" (UniqueName: \"kubernetes.io/projected/240deed4-560c-47ca-a440-2477741fec13-kube-api-access-qz5qq\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.531276 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.531242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/240deed4-560c-47ca-a440-2477741fec13-signing-key\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.531436 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.531361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/240deed4-560c-47ca-a440-2477741fec13-signing-cabundle\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.587598 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.587566    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gf5ph_31c2a323-206c-4f9c-b65c-555120aa0507/node-ca/0.log"
Jul 28 00:46:56.632520 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.632487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/240deed4-560c-47ca-a440-2477741fec13-signing-cabundle\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.632670 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.632588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5qq\" (UniqueName: \"kubernetes.io/projected/240deed4-560c-47ca-a440-2477741fec13-kube-api-access-qz5qq\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.632670 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.632617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/240deed4-560c-47ca-a440-2477741fec13-signing-key\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.633270 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.633246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/240deed4-560c-47ca-a440-2477741fec13-signing-cabundle\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.634897 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.634871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/240deed4-560c-47ca-a440-2477741fec13-signing-key\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.641440 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.641416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5qq\" (UniqueName: \"kubernetes.io/projected/240deed4-560c-47ca-a440-2477741fec13-kube-api-access-qz5qq\") pod \"service-ca-658d7b7c78-cv8g9\" (UID: \"240deed4-560c-47ca-a440-2477741fec13\") " pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.762638 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.762613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9"
Jul 28 00:46:56.922753 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.922616    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-cv8g9"]
Jul 28 00:46:56.925227 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:46:56.925200    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240deed4_560c_47ca_a440_2477741fec13.slice/crio-9472a85727b17f8474068e78128362803caddac206e026251dbd40c519bce18e WatchSource:0}: Error finding container 9472a85727b17f8474068e78128362803caddac206e026251dbd40c519bce18e: Status 404 returned error can't find the container with id 9472a85727b17f8474068e78128362803caddac206e026251dbd40c519bce18e
Jul 28 00:46:56.988857 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:56.988830    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrvdd_493749e8-7d19-468f-a472-fceb6face574/serve-healthcheck-canary/0.log"
Jul 28 00:46:57.244955 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.244897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9" event={"ID":"240deed4-560c-47ca-a440-2477741fec13","Type":"ContainerStarted","Data":"37f90fa6e6a5f6b05b2115f80349b7402008b5b89c2258db4a3af1f336e902d7"}
Jul 28 00:46:57.245392 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.244971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9" event={"ID":"240deed4-560c-47ca-a440-2477741fec13","Type":"ContainerStarted","Data":"9472a85727b17f8474068e78128362803caddac206e026251dbd40c519bce18e"}
Jul 28 00:46:57.247027 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.247004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4" event={"ID":"a1b37d41-216e-418d-a163-e238c70acb9e","Type":"ContainerStarted","Data":"4487f8e16fc617b2b6344000ded8c6c789d6be9355d4221bd334d4d9fb11e3cf"}
Jul 28 00:46:57.247142 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.247050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4" event={"ID":"a1b37d41-216e-418d-a163-e238c70acb9e","Type":"ContainerStarted","Data":"ef0cbd5bdc44a2a3cb936162d73f4c285fd41fecc58c5e1bb73398e8bd0fc433"}
Jul 28 00:46:57.263555 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.263506    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-cv8g9" podStartSLOduration=1.263492268 podStartE2EDuration="1.263492268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:57.262004506 +0000 UTC m=+48.817507682" watchObservedRunningTime="2026-07-28 00:46:57.263492268 +0000 UTC m=+48.818995428"
Jul 28 00:46:57.283819 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.283770    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-ggxp4" podStartSLOduration=1.828267361 podStartE2EDuration="3.283755579s" totalImagesPullingTime="1.455488218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:57.282850003 +0000 UTC m=+48.838353163" watchObservedRunningTime="2026-07-28 00:46:57.283755579 +0000 UTC m=+48.839258775"
Jul 28 00:46:57.325268 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.325227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:57.945992 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.945949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:57.946148 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:57.946008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:46:57.946148 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:57.946103    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 00:46:57.946148 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:57.946103    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:57.946263 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:57.946157    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:01.94614325 +0000 UTC m=+53.501646388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : configmap "kube-rbac-proxy" not found
Jul 28 00:46:57.946263 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:57.946194    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:01.946178176 +0000 UTC m=+53.501681318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : secret "insights-runtime-extractor-tls" not found
Jul 28 00:46:59.260553 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:46:59.260553 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260693    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260721    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:15.260700569 +0000 UTC m=+66.816203716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : configmap references non-existent config key: service-ca.crt
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260759    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls podName:61ce6a7c-6678-4431-a4eb-6ceacba6faef nodeName:}" failed. No retries permitted until 2026-07-28 00:47:15.26073979 +0000 UTC m=+66.816242928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls") pod "dns-default-7vx6j" (UID: "61ce6a7c-6678-4431-a4eb-6ceacba6faef") : secret "dns-default-metrics-tls" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.260845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260765    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260918    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls podName:05ad823c-b798-4bc0-95fe-2885ce46e53b nodeName:}" failed. No retries permitted until 2026-07-28 00:47:15.260905863 +0000 UTC m=+66.816409021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9jrqd" (UID: "05ad823c-b798-4bc0-95fe-2885ce46e53b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260814    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260971    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs podName:ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:15.260962369 +0000 UTC m=+66.816465509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs") pod "router-default-d48b877f-xhtsx" (UID: "ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5") : secret "router-metrics-certs-default" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260872    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260980    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 00:46:59.261115 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.260991    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbff6fc5-m2lf9: secret "image-registry-tls" not found
Jul 28 00:46:59.261639 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.261022    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert podName:29f05ced-501b-4e37-b673-706d1a80b7af nodeName:}" failed. No retries permitted until 2026-07-28 00:47:15.261010982 +0000 UTC m=+66.816514122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v5ct8" (UID: "29f05ced-501b-4e37-b673-706d1a80b7af") : secret "networking-console-plugin-cert" not found
Jul 28 00:46:59.261639 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:46:59.261040    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls podName:f0660dbe-4d29-4a5c-84b2-6e25b6af7603 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:15.261030476 +0000 UTC m=+66.816533615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls") pod "image-registry-69dbff6fc5-m2lf9" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603") : secret "image-registry-tls" not found
Jul 28 00:46:59.263236 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.263211    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9024cb5-e372-4d36-84f4-d9935078d587-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xj2zm\" (UID: \"c9024cb5-e372-4d36-84f4-d9935078d587\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:59.548889 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.548811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"
Jul 28 00:46:59.695117 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:46:59.695087    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm"]
Jul 28 00:47:00.260687 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:00.260642    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm" event={"ID":"c9024cb5-e372-4d36-84f4-d9935078d587","Type":"ContainerStarted","Data":"8de90ee1f4ed8d5d1e1345684a82cd598f813a4837f26bd59a226aa5784d7994"}
Jul 28 00:47:01.982862 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:01.982825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:47:01.983324 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:01.982887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:47:01.983324 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:47:01.982964    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 00:47:01.983324 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:47:01.983048    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls podName:6bcdb825-e508-4e01-ad7b-65915bbd8b91 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:09.983027238 +0000 UTC m=+61.538530391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5hrw" (UID: "6bcdb825-e508-4e01-ad7b-65915bbd8b91") : secret "insights-runtime-extractor-tls" not found
Jul 28 00:47:01.983480 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:01.983443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bcdb825-e508-4e01-ad7b-65915bbd8b91-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:47:02.269547 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:02.269513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm" event={"ID":"c9024cb5-e372-4d36-84f4-d9935078d587","Type":"ContainerStarted","Data":"2cfcc6ff18e270bdc27c64f4050065f8abae4783f2dfd696108a6017997efb3f"}
Jul 28 00:47:02.269729 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:02.269592    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm" event={"ID":"c9024cb5-e372-4d36-84f4-d9935078d587","Type":"ContainerStarted","Data":"fe4b6cf93137326ee9b4ade50da96572f3e84355c34b3e61c4d523258a64f856"}
Jul 28 00:47:02.284012 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:02.283964    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xj2zm" podStartSLOduration=51.267372129 podStartE2EDuration="53.283924145s" totalImagesPullingTime="2.016552016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:02.283482478 +0000 UTC m=+53.838985651" watchObservedRunningTime="2026-07-28 00:47:02.283924145 +0000 UTC m=+53.839427284"
Jul 28 00:47:03.218282 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:03.218241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:47:03.218728 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:03.218629    2564 scope.go:117] "RemoveContainer" containerID="7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785"
Jul 28 00:47:03.218800 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:47:03.218779    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-l64k4_openshift-console-operator(b707cfdd-c2b5-4dc8-b386-b096fb2335c1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1"
Jul 28 00:47:03.706643 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:03.706603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:47:03.706994 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:03.706981    2564 scope.go:117] "RemoveContainer" containerID="7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785"
Jul 28 00:47:03.707141 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:47:03.707124    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-l64k4_openshift-console-operator(b707cfdd-c2b5-4dc8-b386-b096fb2335c1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1"
Jul 28 00:47:09.137604 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:09.137573    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsmjh"
Jul 28 00:47:10.064689 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:10.064648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:47:10.066997 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:10.066973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bcdb825-e508-4e01-ad7b-65915bbd8b91-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5hrw\" (UID: \"6bcdb825-e508-4e01-ad7b-65915bbd8b91\") " pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:47:10.139637 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:10.139600    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f27qr\""
Jul 28 00:47:10.148505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:10.148483    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q5hrw"
Jul 28 00:47:10.273975 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:10.273904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q5hrw"]
Jul 28 00:47:10.276572 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:10.276544    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcdb825_e508_4e01_ad7b_65915bbd8b91.slice/crio-21dd1fe7181fdd51282066da7f2866f2b342e11f648dc88a8ccbf8649bc164b9 WatchSource:0}: Error finding container 21dd1fe7181fdd51282066da7f2866f2b342e11f648dc88a8ccbf8649bc164b9: Status 404 returned error can't find the container with id 21dd1fe7181fdd51282066da7f2866f2b342e11f648dc88a8ccbf8649bc164b9
Jul 28 00:47:10.297857 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:10.297826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5hrw" event={"ID":"6bcdb825-e508-4e01-ad7b-65915bbd8b91","Type":"ContainerStarted","Data":"21dd1fe7181fdd51282066da7f2866f2b342e11f648dc88a8ccbf8649bc164b9"}
Jul 28 00:47:14.612339 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.612302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:47:14.612862 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.612425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:47:14.614990 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.614971    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 00:47:14.615542 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.615520    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 00:47:14.625536 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.625512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da18fd8-305e-4996-a531-2f569665d052-metrics-certs\") pod \"network-metrics-daemon-g88nz\" (UID: \"0da18fd8-305e-4996-a531-2f569665d052\") " pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:47:14.625663 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.625515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8-original-pull-secret\") pod \"global-pull-secret-syncer-6vvw7\" (UID: \"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8\") " pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:47:14.659860 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.659832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kdcl9\""
Jul 28 00:47:14.667591 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.667567    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g88nz"
Jul 28 00:47:14.814284 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.814247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:47:14.816582 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.816552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgcxk\" (UniqueName: \"kubernetes.io/projected/2640711a-54c5-435f-a218-e89af0639bcb-kube-api-access-bgcxk\") pod \"network-check-target-z5mrk\" (UID: \"2640711a-54c5-435f-a218-e89af0639bcb\") " pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:47:14.859536 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.859502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vvw7"
Jul 28 00:47:14.939397 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.939368    2564 scope.go:117] "RemoveContainer" containerID="7f60b71bf9eeceaede543a450555ddebb0fef97e25790212f4689c7b55107785"
Jul 28 00:47:14.954234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.954206    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkfth\""
Jul 28 00:47:14.962975 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:14.962942    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:47:15.322750 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.322673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z5mrk"]
Jul 28 00:47:15.323208 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.323074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:15.323208 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.323122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:47:15.323208 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.323168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:47:15.323444 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.323223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:47:15.323444 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.323250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:15.323444 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.323296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:15.327857 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.327743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-service-ca-bundle\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:15.329375 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.329351    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2640711a_54c5_435f_a218_e89af0639bcb.slice/crio-c81ad4482e22427f438da2dac7398f67f2672af960e513d733ae5c0a7160a067 WatchSource:0}: Error finding container c81ad4482e22427f438da2dac7398f67f2672af960e513d733ae5c0a7160a067: Status 404 returned error can't find the container with id c81ad4482e22427f438da2dac7398f67f2672af960e513d733ae5c0a7160a067
Jul 28 00:47:15.330781 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.330462    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f05ced-501b-4e37-b673-706d1a80b7af-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v5ct8\" (UID: \"29f05ced-501b-4e37-b673-706d1a80b7af\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:47:15.330781 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.330717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5hrw" event={"ID":"6bcdb825-e508-4e01-ad7b-65915bbd8b91","Type":"ContainerStarted","Data":"e3889770b4d0535ca8c1c891e7b170e56bf810e2ed95b98a0c91d5dc321e1d7f"}
Jul 28 00:47:15.331056 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.331032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5-metrics-certs\") pod \"router-default-d48b877f-xhtsx\" (UID: \"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5\") " pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:15.332594 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.332365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ce6a7c-6678-4431-a4eb-6ceacba6faef-metrics-tls\") pod \"dns-default-7vx6j\" (UID: \"61ce6a7c-6678-4431-a4eb-6ceacba6faef\") " pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:47:15.332594 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.332391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"image-registry-69dbff6fc5-m2lf9\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") " pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:15.333662 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.333633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/05ad823c-b798-4bc0-95fe-2885ce46e53b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9jrqd\" (UID: \"05ad823c-b798-4bc0-95fe-2885ce46e53b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:47:15.334178 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.334157    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:47:15.334779 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.334523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" event={"ID":"b707cfdd-c2b5-4dc8-b386-b096fb2335c1","Type":"ContainerStarted","Data":"6aafc44eb0fe1ea888defd6214beaa1a6b9ec4f20f7849186d31e399ce3f744e"}
Jul 28 00:47:15.343394 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.336169    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:47:15.343394 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.343182    2564 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-l64k4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Jul 28 00:47:15.343394 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.343241    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4" podUID="b707cfdd-c2b5-4dc8-b386-b096fb2335c1" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Jul 28 00:47:15.345481 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.345461    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g88nz"]
Jul 28 00:47:15.348379 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.348355    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da18fd8_305e_4996_a531_2f569665d052.slice/crio-533b84ec208e519342b19ccd3663702544e654abc4d1a1611e32add784154067 WatchSource:0}: Error finding container 533b84ec208e519342b19ccd3663702544e654abc4d1a1611e32add784154067: Status 404 returned error can't find the container with id 533b84ec208e519342b19ccd3663702544e654abc4d1a1611e32add784154067
Jul 28 00:47:15.367392 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.367368    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vvw7"]
Jul 28 00:47:15.370205 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.370184    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5c29c5_6324_44b2_ad00_5d8c4b9a36e8.slice/crio-897768814f88073cf30c08ab07ddecd5cb052ffcfc41e8317ec1c3c6f1e72368 WatchSource:0}: Error finding container 897768814f88073cf30c08ab07ddecd5cb052ffcfc41e8317ec1c3c6f1e72368: Status 404 returned error can't find the container with id 897768814f88073cf30c08ab07ddecd5cb052ffcfc41e8317ec1c3c6f1e72368
Jul 28 00:47:15.462391 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.462325    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rnvgz\""
Jul 28 00:47:15.470531 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.470513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"
Jul 28 00:47:15.491800 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.491776    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-54rrh\""
Jul 28 00:47:15.500734 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.500708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:47:15.515444 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.515348    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bvhj4\""
Jul 28 00:47:15.523176 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.523147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:15.529508 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.529280    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qz9w8\""
Jul 28 00:47:15.537226 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.536887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:15.541493 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.541466    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h2f8w\""
Jul 28 00:47:15.546755 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.546730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"
Jul 28 00:47:15.641697 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.641669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd"]
Jul 28 00:47:15.645837 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.645786    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ad823c_b798_4bc0_95fe_2885ce46e53b.slice/crio-e790e216adc79cc8c6a6446fdeb07bcaf4dd1c59767d64bafdf77e1f0ae319da WatchSource:0}: Error finding container e790e216adc79cc8c6a6446fdeb07bcaf4dd1c59767d64bafdf77e1f0ae319da: Status 404 returned error can't find the container with id e790e216adc79cc8c6a6446fdeb07bcaf4dd1c59767d64bafdf77e1f0ae319da
Jul 28 00:47:15.706950 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.706523    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7vx6j"]
Jul 28 00:47:15.716626 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.716328    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ce6a7c_6678_4431_a4eb_6ceacba6faef.slice/crio-f140bca1239585db49bcf94d7d048656230435ae174ad15e071088c333efd879 WatchSource:0}: Error finding container f140bca1239585db49bcf94d7d048656230435ae174ad15e071088c333efd879: Status 404 returned error can't find the container with id f140bca1239585db49bcf94d7d048656230435ae174ad15e071088c333efd879
Jul 28 00:47:15.749123 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.749003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d48b877f-xhtsx"]
Jul 28 00:47:15.766811 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.766690    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69dbff6fc5-m2lf9"]
Jul 28 00:47:15.779763 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.779707    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0660dbe_4d29_4a5c_84b2_6e25b6af7603.slice/crio-e44bd818328a61cd39ef7c3582465f9c662d5a8cf48a5437349720256c2e94fd WatchSource:0}: Error finding container e44bd818328a61cd39ef7c3582465f9c662d5a8cf48a5437349720256c2e94fd: Status 404 returned error can't find the container with id e44bd818328a61cd39ef7c3582465f9c662d5a8cf48a5437349720256c2e94fd
Jul 28 00:47:15.791645 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:15.791620    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8"]
Jul 28 00:47:15.794928 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:15.794898    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f05ced_501b_4e37_b673_706d1a80b7af.slice/crio-0552f407d799f1cb76323dfb34cb144ce1c8a806e96783c81ca1ace655b9f79c WatchSource:0}: Error finding container 0552f407d799f1cb76323dfb34cb144ce1c8a806e96783c81ca1ace655b9f79c: Status 404 returned error can't find the container with id 0552f407d799f1cb76323dfb34cb144ce1c8a806e96783c81ca1ace655b9f79c
Jul 28 00:47:16.356098 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.356059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d48b877f-xhtsx" event={"ID":"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5","Type":"ContainerStarted","Data":"4d0cb29cd5a173961f128f4c1ef11a868e2e05a8d0a59643e7b81dc055c6c432"}
Jul 28 00:47:16.356293 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.356125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d48b877f-xhtsx" event={"ID":"ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5","Type":"ContainerStarted","Data":"f5042ea8b5fb73f76c4102aaad9aa1d51ef7d32ae03d3dadc9f2720847670b62"}
Jul 28 00:47:16.362977 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.362926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vvw7" event={"ID":"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8","Type":"ContainerStarted","Data":"897768814f88073cf30c08ab07ddecd5cb052ffcfc41e8317ec1c3c6f1e72368"}
Jul 28 00:47:16.368862 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.368829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g88nz" event={"ID":"0da18fd8-305e-4996-a531-2f569665d052","Type":"ContainerStarted","Data":"533b84ec208e519342b19ccd3663702544e654abc4d1a1611e32add784154067"}
Jul 28 00:47:16.373916 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.373837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8" event={"ID":"29f05ced-501b-4e37-b673-706d1a80b7af","Type":"ContainerStarted","Data":"0552f407d799f1cb76323dfb34cb144ce1c8a806e96783c81ca1ace655b9f79c"}
Jul 28 00:47:16.380695 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.380633    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d48b877f-xhtsx" podStartSLOduration=67.38061167 podStartE2EDuration="1m7.38061167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:16.378544182 +0000 UTC m=+67.934047360" watchObservedRunningTime="2026-07-28 00:47:16.38061167 +0000 UTC m=+67.936114834"
Jul 28 00:47:16.384448 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.383571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9" event={"ID":"f0660dbe-4d29-4a5c-84b2-6e25b6af7603","Type":"ContainerStarted","Data":"659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7"}
Jul 28 00:47:16.384448 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.383623    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9" event={"ID":"f0660dbe-4d29-4a5c-84b2-6e25b6af7603","Type":"ContainerStarted","Data":"e44bd818328a61cd39ef7c3582465f9c662d5a8cf48a5437349720256c2e94fd"}
Jul 28 00:47:16.384448 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.384400    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:16.390982 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.390885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7vx6j" event={"ID":"61ce6a7c-6678-4431-a4eb-6ceacba6faef","Type":"ContainerStarted","Data":"f140bca1239585db49bcf94d7d048656230435ae174ad15e071088c333efd879"}
Jul 28 00:47:16.393167 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.393093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z5mrk" event={"ID":"2640711a-54c5-435f-a218-e89af0639bcb","Type":"ContainerStarted","Data":"471d40345e20da21e4776bece446f9026dbd3dcdca1582187e5b2e9a6ec7e9c7"}
Jul 28 00:47:16.393167 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.393133    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z5mrk" event={"ID":"2640711a-54c5-435f-a218-e89af0639bcb","Type":"ContainerStarted","Data":"c81ad4482e22427f438da2dac7398f67f2672af960e513d733ae5c0a7160a067"}
Jul 28 00:47:16.393417 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.393380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:47:16.395352 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.395306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd" event={"ID":"05ad823c-b798-4bc0-95fe-2885ce46e53b","Type":"ContainerStarted","Data":"e790e216adc79cc8c6a6446fdeb07bcaf4dd1c59767d64bafdf77e1f0ae319da"}
Jul 28 00:47:16.402722 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.402670    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-l64k4"
Jul 28 00:47:16.405320 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.404159    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9" podStartSLOduration=54.404140493 podStartE2EDuration="54.404140493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:16.403204183 +0000 UTC m=+67.958707405" watchObservedRunningTime="2026-07-28 00:47:16.404140493 +0000 UTC m=+67.959643655"
Jul 28 00:47:16.437901 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.437449    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z5mrk" podStartSLOduration=67.437426996 podStartE2EDuration="1m7.437426996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:16.436499397 +0000 UTC m=+67.992002559" watchObservedRunningTime="2026-07-28 00:47:16.437426996 +0000 UTC m=+67.992930157"
Jul 28 00:47:16.524963 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.524468    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:16.528113 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.527884    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:16.852187 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.852146    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"]
Jul 28 00:47:16.870917 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.870884    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"]
Jul 28 00:47:16.872494 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.872454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:16.875306 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.875282    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 00:47:16.875704 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.875682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 00:47:16.876670 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.875928    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 00:47:16.876670 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.876224    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 00:47:16.886805 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.886043    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"]
Jul 28 00:47:16.888488 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.887623    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:16.892228 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.891320    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 00:47:16.892228 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.891703    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 00:47:16.892228 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.891878    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 00:47:16.892228 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.892073    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 00:47:16.902968 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.902004    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"]
Jul 28 00:47:16.902968 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.902068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"]
Jul 28 00:47:16.902968 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.902085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"]
Jul 28 00:47:16.902968 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.902108    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:16.904635 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.904615    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8mzqv\""
Jul 28 00:47:16.905024 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.904999    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 00:47:16.915541 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.915511    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69dbff6fc5-m2lf9"]
Jul 28 00:47:16.953923 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.953896    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-88wn4"]
Jul 28 00:47:16.979958 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.979908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-88wn4"]
Jul 28 00:47:16.980244 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.980201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:16.983717 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.983512    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 00:47:16.984131 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.984067    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 00:47:16.984481 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:16.984459    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6d6mx\""
Jul 28 00:47:17.040353 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040315    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jwp\" (UniqueName: \"kubernetes.io/projected/343e3aeb-34e3-4488-8461-16b6c958307c-kube-api-access-m6jwp\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/343e3aeb-34e3-4488-8461-16b6c958307c-tmp\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/986d0933-10d9-4fba-937e-9995301c208d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-hub\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/343e3aeb-34e3-4488-8461-16b6c958307c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xvf\" (UniqueName: \"kubernetes.io/projected/986d0933-10d9-4fba-937e-9995301c208d-kube-api-access-74xvf\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9gg\" (UniqueName: \"kubernetes.io/projected/fb459d7c-676e-44ae-8a02-f09cf6aec680-kube-api-access-9n9gg\") pod \"managed-serviceaccount-addon-agent-578fc94fb4-gjxlg\" (UID: \"fb459d7c-676e-44ae-8a02-f09cf6aec680\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.040622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb459d7c-676e-44ae-8a02-f09cf6aec680-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578fc94fb4-gjxlg\" (UID: \"fb459d7c-676e-44ae-8a02-f09cf6aec680\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.041130 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-ca\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.041130 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.040682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.046485 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.046457    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d559cd4cf-d258t"]
Jul 28 00:47:17.064520 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.064423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d559cd4cf-d258t"]
Jul 28 00:47:17.064646 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.064453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/343e3aeb-34e3-4488-8461-16b6c958307c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143770    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xvf\" (UniqueName: \"kubernetes.io/projected/986d0933-10d9-4fba-937e-9995301c208d-kube-api-access-74xvf\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9gg\" (UniqueName: \"kubernetes.io/projected/fb459d7c-676e-44ae-8a02-f09cf6aec680-kube-api-access-9n9gg\") pod \"managed-serviceaccount-addon-agent-578fc94fb4-gjxlg\" (UID: \"fb459d7c-676e-44ae-8a02-f09cf6aec680\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb459d7c-676e-44ae-8a02-f09cf6aec680-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578fc94fb4-gjxlg\" (UID: \"fb459d7c-676e-44ae-8a02-f09cf6aec680\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-ca\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddzs\" (UniqueName: \"kubernetes.io/projected/6b384be7-70d1-47a4-84be-fe3f3c8a2617-kube-api-access-nddzs\") pod \"downloads-7c97dc4555-88wn4\" (UID: \"6b384be7-70d1-47a4-84be-fe3f3c8a2617\") " pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.143999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jwp\" (UniqueName: \"kubernetes.io/projected/343e3aeb-34e3-4488-8461-16b6c958307c-kube-api-access-m6jwp\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.144027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/343e3aeb-34e3-4488-8461-16b6c958307c-tmp\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.144044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/986d0933-10d9-4fba-937e-9995301c208d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.144467 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.144069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-hub\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.145963 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.145857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/343e3aeb-34e3-4488-8461-16b6c958307c-tmp\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.146599 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.146572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/986d0933-10d9-4fba-937e-9995301c208d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.147793 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.147715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-ca\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.151138 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.150213    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/343e3aeb-34e3-4488-8461-16b6c958307c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.151138 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.150867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.151519 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.151472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-hub\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.151949 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.151912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/986d0933-10d9-4fba-937e-9995301c208d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.153311 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.153273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb459d7c-676e-44ae-8a02-f09cf6aec680-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578fc94fb4-gjxlg\" (UID: \"fb459d7c-676e-44ae-8a02-f09cf6aec680\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.158890 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.158373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jwp\" (UniqueName: \"kubernetes.io/projected/343e3aeb-34e3-4488-8461-16b6c958307c-kube-api-access-m6jwp\") pod \"klusterlet-addon-workmgr-7bf5b6b4b9-kqpns\" (UID: \"343e3aeb-34e3-4488-8461-16b6c958307c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.164835 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.164780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9gg\" (UniqueName: \"kubernetes.io/projected/fb459d7c-676e-44ae-8a02-f09cf6aec680-kube-api-access-9n9gg\") pod \"managed-serviceaccount-addon-agent-578fc94fb4-gjxlg\" (UID: \"fb459d7c-676e-44ae-8a02-f09cf6aec680\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.168588 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.168531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xvf\" (UniqueName: \"kubernetes.io/projected/986d0933-10d9-4fba-937e-9995301c208d-kube-api-access-74xvf\") pod \"cluster-proxy-proxy-agent-5bb65956dd-kknzs\" (UID: \"986d0933-10d9-4fba-937e-9995301c208d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.193686 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.193647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:17.220296 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.219832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"
Jul 28 00:47:17.230195 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.230112    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.244984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77f8q\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-kube-api-access-77f8q\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4d208d-719d-4648-9422-0f394cc5463d-trusted-ca\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4d208d-719d-4648-9422-0f394cc5463d-image-registry-private-configuration\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-registry-tls\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-bound-sa-token\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddzs\" (UniqueName: \"kubernetes.io/projected/6b384be7-70d1-47a4-84be-fe3f3c8a2617-kube-api-access-nddzs\") pod \"downloads-7c97dc4555-88wn4\" (UID: \"6b384be7-70d1-47a4-84be-fe3f3c8a2617\") " pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4d208d-719d-4648-9422-0f394cc5463d-ca-trust-extracted\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4d208d-719d-4648-9422-0f394cc5463d-installation-pull-secrets\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.245533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.245341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4d208d-719d-4648-9422-0f394cc5463d-registry-certificates\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.257396 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.257344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddzs\" (UniqueName: \"kubernetes.io/projected/6b384be7-70d1-47a4-84be-fe3f3c8a2617-kube-api-access-nddzs\") pod \"downloads-7c97dc4555-88wn4\" (UID: \"6b384be7-70d1-47a4-84be-fe3f3c8a2617\") " pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:17.302323 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.302116    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4d208d-719d-4648-9422-0f394cc5463d-trusted-ca\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4d208d-719d-4648-9422-0f394cc5463d-image-registry-private-configuration\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-registry-tls\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-bound-sa-token\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4d208d-719d-4648-9422-0f394cc5463d-ca-trust-extracted\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4d208d-719d-4648-9422-0f394cc5463d-installation-pull-secrets\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346263    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4d208d-719d-4648-9422-0f394cc5463d-registry-certificates\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.346327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f8q\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-kube-api-access-77f8q\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.347619    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4d208d-719d-4648-9422-0f394cc5463d-trusted-ca\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.348505 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.347895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae4d208d-719d-4648-9422-0f394cc5463d-ca-trust-extracted\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.349360 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.349304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae4d208d-719d-4648-9422-0f394cc5463d-registry-certificates\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.351663 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.351476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae4d208d-719d-4648-9422-0f394cc5463d-image-registry-private-configuration\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.353702 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.353640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-registry-tls\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.355826 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.355781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae4d208d-719d-4648-9422-0f394cc5463d-installation-pull-secrets\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.356393 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.356368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f8q\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-kube-api-access-77f8q\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.357327 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.357284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4d208d-719d-4648-9422-0f394cc5463d-bound-sa-token\") pod \"image-registry-5d559cd4cf-d258t\" (UID: \"ae4d208d-719d-4648-9422-0f394cc5463d\") " pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.379191 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.378930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:17.403374 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.403275    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:17.405487 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:17.405259    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d48b877f-xhtsx"
Jul 28 00:47:19.763975 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:19.763928    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-88wn4"]
Jul 28 00:47:22.648142 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:22.648108    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b384be7_70d1_47a4_84be_fe3f3c8a2617.slice/crio-77861f333f02b7f98c779cd680307586f967be24e29f47bfba184b43435d5beb WatchSource:0}: Error finding container 77861f333f02b7f98c779cd680307586f967be24e29f47bfba184b43435d5beb: Status 404 returned error can't find the container with id 77861f333f02b7f98c779cd680307586f967be24e29f47bfba184b43435d5beb
Jul 28 00:47:22.774623 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:22.774594    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"]
Jul 28 00:47:22.957974 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:22.957902    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343e3aeb_34e3_4488_8461_16b6c958307c.slice/crio-7d37902ecd10c67ff520a9b3c44d8a7cb1e0c294cac4b9647953ea74f057bf48 WatchSource:0}: Error finding container 7d37902ecd10c67ff520a9b3c44d8a7cb1e0c294cac4b9647953ea74f057bf48: Status 404 returned error can't find the container with id 7d37902ecd10c67ff520a9b3c44d8a7cb1e0c294cac4b9647953ea74f057bf48
Jul 28 00:47:23.087963 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:23.087921    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg"]
Jul 28 00:47:23.427045 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:23.426961    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns" event={"ID":"343e3aeb-34e3-4488-8461-16b6c958307c","Type":"ContainerStarted","Data":"7d37902ecd10c67ff520a9b3c44d8a7cb1e0c294cac4b9647953ea74f057bf48"}
Jul 28 00:47:23.428292 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:23.428266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-88wn4" event={"ID":"6b384be7-70d1-47a4-84be-fe3f3c8a2617","Type":"ContainerStarted","Data":"77861f333f02b7f98c779cd680307586f967be24e29f47bfba184b43435d5beb"}
Jul 28 00:47:23.674148 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:23.673706    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb459d7c_676e_44ae_8a02_f09cf6aec680.slice/crio-7e36886c3fb4db517b688a3e6e0982c7f9c428b169019c63e0ae482f1b98566a WatchSource:0}: Error finding container 7e36886c3fb4db517b688a3e6e0982c7f9c428b169019c63e0ae482f1b98566a: Status 404 returned error can't find the container with id 7e36886c3fb4db517b688a3e6e0982c7f9c428b169019c63e0ae482f1b98566a
Jul 28 00:47:23.809323 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:23.809293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d559cd4cf-d258t"]
Jul 28 00:47:23.844446 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:23.844101    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4d208d_719d_4648_9422_0f394cc5463d.slice/crio-79c3ae914cede0678caea8195a9b53c4a2bbaaa8d0f3120bb44efc9bc3c16285 WatchSource:0}: Error finding container 79c3ae914cede0678caea8195a9b53c4a2bbaaa8d0f3120bb44efc9bc3c16285: Status 404 returned error can't find the container with id 79c3ae914cede0678caea8195a9b53c4a2bbaaa8d0f3120bb44efc9bc3c16285
Jul 28 00:47:24.034130 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.034074    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs"]
Jul 28 00:47:24.456703 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.456525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8" event={"ID":"29f05ced-501b-4e37-b673-706d1a80b7af","Type":"ContainerStarted","Data":"bac0d176fe3f1c8e671d7858be26f0120909ea261c4e996dd0f59f9b69ec57cb"}
Jul 28 00:47:24.466696 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.466604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7vx6j" event={"ID":"61ce6a7c-6678-4431-a4eb-6ceacba6faef","Type":"ContainerStarted","Data":"4e6664c2ec03b35131a5025c49c9610447aa42c3fbb5a4ad6666f8421482db14"}
Jul 28 00:47:24.480751 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.478874    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v5ct8" podStartSLOduration=35.438358933 podStartE2EDuration="43.478856153s" totalImagesPullingTime="8.04049722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:24.477782326 +0000 UTC m=+76.033285527" watchObservedRunningTime="2026-07-28 00:47:24.478856153 +0000 UTC m=+76.034359314"
Jul 28 00:47:24.480751 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.478906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd" event={"ID":"05ad823c-b798-4bc0-95fe-2885ce46e53b","Type":"ContainerStarted","Data":"fbe484495311427e0660c28451d5aacf610edccbedd23ad11450050c8883a43b"}
Jul 28 00:47:24.493051 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.492342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vvw7" event={"ID":"8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8","Type":"ContainerStarted","Data":"daff7754e2afef5d30a4ca3d21cbaee882e3d91e6e063525db76d6205ae007d5"}
Jul 28 00:47:24.510689 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.508695    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9jrqd" podStartSLOduration=67.326948586 podStartE2EDuration="1m15.50867468s" totalImagesPullingTime="8.181726094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:24.501811187 +0000 UTC m=+76.057314349" watchObservedRunningTime="2026-07-28 00:47:24.50867468 +0000 UTC m=+76.064177842"
Jul 28 00:47:24.510689 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.509668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t" event={"ID":"ae4d208d-719d-4648-9422-0f394cc5463d","Type":"ContainerStarted","Data":"2c06ece6a0e690e88e7ab475014428549a00efe85c8eb7bb41a361842ae1c118"}
Jul 28 00:47:24.510689 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.509731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t" event={"ID":"ae4d208d-719d-4648-9422-0f394cc5463d","Type":"ContainerStarted","Data":"79c3ae914cede0678caea8195a9b53c4a2bbaaa8d0f3120bb44efc9bc3c16285"}
Jul 28 00:47:24.510689 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.510368    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:24.540058 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.539516    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6vvw7" podStartSLOduration=67.063548678 podStartE2EDuration="1m15.539494955s" totalImagesPullingTime="8.475946277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:24.537709426 +0000 UTC m=+76.093212608" watchObservedRunningTime="2026-07-28 00:47:24.539494955 +0000 UTC m=+76.094998116"
Jul 28 00:47:24.554842 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.554752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g88nz" event={"ID":"0da18fd8-305e-4996-a531-2f569665d052","Type":"ContainerStarted","Data":"de8ff206e6fb3c228bb2e44a8741a78786c1e63dc7699050b61ad84b12615003"}
Jul 28 00:47:24.565011 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.564956    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t" podStartSLOduration=7.564919872 podStartE2EDuration="7.564919872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:24.562192125 +0000 UTC m=+76.117695310" watchObservedRunningTime="2026-07-28 00:47:24.564919872 +0000 UTC m=+76.120423033"
Jul 28 00:47:24.566605 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.566498    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs" event={"ID":"986d0933-10d9-4fba-937e-9995301c208d","Type":"ContainerStarted","Data":"a9d29b80b0650673c26c0d8a5c418c319ff02ae787c95723e249dcbc8a4496f3"}
Jul 28 00:47:24.578230 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:24.575672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg" event={"ID":"fb459d7c-676e-44ae-8a02-f09cf6aec680","Type":"ContainerStarted","Data":"7e36886c3fb4db517b688a3e6e0982c7f9c428b169019c63e0ae482f1b98566a"}
Jul 28 00:47:25.605586 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:25.605459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5hrw" event={"ID":"6bcdb825-e508-4e01-ad7b-65915bbd8b91","Type":"ContainerStarted","Data":"20bb59651dde2fae810a0ed9ccdf1f860a4ab7a058d72f4924889348c6ad1d00"}
Jul 28 00:47:25.615603 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:25.614638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g88nz" event={"ID":"0da18fd8-305e-4996-a531-2f569665d052","Type":"ContainerStarted","Data":"ba900d582f70aad8076ab99f7453b00c16f745abca1ddb14c64dde6c9e44e686"}
Jul 28 00:47:25.634587 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:25.634522    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g88nz" podStartSLOduration=70.842420074 podStartE2EDuration="1m16.634500917s" totalImagesPullingTime="5.792080843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:25.633393894 +0000 UTC m=+77.188897079" watchObservedRunningTime="2026-07-28 00:47:25.634500917 +0000 UTC m=+77.190004078"
Jul 28 00:47:25.640295 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:25.640261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7vx6j" event={"ID":"61ce6a7c-6678-4431-a4eb-6ceacba6faef","Type":"ContainerStarted","Data":"2450553c4c2390618fa82d6f7e3a3c95f76bab5ac9963320fc99e054b4a0a5ad"}
Jul 28 00:47:25.641497 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:25.641471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:47:34.706200 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.706156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5hrw" event={"ID":"6bcdb825-e508-4e01-ad7b-65915bbd8b91","Type":"ContainerStarted","Data":"d038eace6134baa2a7af1835794db823c29bae4cb53c9a7337ed8d20bf0e9aa7"}
Jul 28 00:47:34.707968 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.707913    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns" event={"ID":"343e3aeb-34e3-4488-8461-16b6c958307c","Type":"ContainerStarted","Data":"eea4b069910585873eeb5d26510b9e887627cede0e46befe5eb902247d496b27"}
Jul 28 00:47:34.708123 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.708101    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:34.709996 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.709973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns"
Jul 28 00:47:34.710533 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.710509    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs" event={"ID":"986d0933-10d9-4fba-937e-9995301c208d","Type":"ContainerStarted","Data":"b608ade312c77f335e11099e609959994ce304b50c4718899f51fb5d372aaf8e"}
Jul 28 00:47:34.710622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.710546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs" event={"ID":"986d0933-10d9-4fba-937e-9995301c208d","Type":"ContainerStarted","Data":"972eb9724f0ebbeb95806f61f07bee9e0375114f36760982048cf8b179c25e0f"}
Jul 28 00:47:34.710622 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.710561    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs" event={"ID":"986d0933-10d9-4fba-937e-9995301c208d","Type":"ContainerStarted","Data":"42f754bdc96e8a2025fdf9fb5d326107bfd9ee91e34afd4c35e982ce9b2c49d5"}
Jul 28 00:47:34.713147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.713119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg" event={"ID":"fb459d7c-676e-44ae-8a02-f09cf6aec680","Type":"ContainerStarted","Data":"1804028e5225e676e18b52b70d99d7fd6aa2b647add139ca26ff59b2369f0875"}
Jul 28 00:47:34.723522 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.723450    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q5hrw" podStartSLOduration=17.604228003 podStartE2EDuration="40.72343685s" totalImagesPullingTime="23.119208847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:34.722087779 +0000 UTC m=+86.277590938" watchObservedRunningTime="2026-07-28 00:47:34.72343685 +0000 UTC m=+86.278939988"
Jul 28 00:47:34.723675 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.723645    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7vx6j" podStartSLOduration=43.602205675 podStartE2EDuration="51.723637619s" totalImagesPullingTime="8.121431944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:25.678843789 +0000 UTC m=+77.234346951" watchObservedRunningTime="2026-07-28 00:47:34.723637619 +0000 UTC m=+86.279140759"
Jul 28 00:47:34.740701 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.740646    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578fc94fb4-gjxlg" podStartSLOduration=8.786000255 podStartE2EDuration="18.740625681s" totalImagesPullingTime="9.954625426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:34.739487196 +0000 UTC m=+86.294990356" watchObservedRunningTime="2026-07-28 00:47:34.740625681 +0000 UTC m=+86.296128842"
Jul 28 00:47:34.756029 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.755983    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bb65956dd-kknzs" podStartSLOduration=9.126107579 podStartE2EDuration="18.755963797s" totalImagesPullingTime="9.629856218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:34.754782989 +0000 UTC m=+86.310286151" watchObservedRunningTime="2026-07-28 00:47:34.755963797 +0000 UTC m=+86.311466954"
Jul 28 00:47:34.769459 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:34.769407    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf5b6b4b9-kqpns" podStartSLOduration=8.065514576 podStartE2EDuration="18.769391998s" totalImagesPullingTime="10.703877422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:34.768498107 +0000 UTC m=+86.324001271" watchObservedRunningTime="2026-07-28 00:47:34.769391998 +0000 UTC m=+86.324895157"
Jul 28 00:47:36.670701 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:36.670666    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7vx6j"
Jul 28 00:47:37.386190 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:37.386153    2564 patch_prober.go:28] interesting pod/image-registry-5d559cd4cf-d258t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 00:47:37.386369 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:37.386220    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t" podUID="ae4d208d-719d-4648-9422-0f394cc5463d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 00:47:38.012623 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.009309    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d4v5w"]
Jul 28 00:47:38.088286 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.088254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.091909 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.091577    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 00:47:38.091909 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.091635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 00:47:38.091909 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.091683    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 00:47:38.091909 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.091585    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 00:47:38.091909 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.091577    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hpvwn\""
Jul 28 00:47:38.153261 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153261 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4br6\" (UniqueName: \"kubernetes.io/projected/e03dd829-a441-43bf-98bd-16395fd459ea-kube-api-access-v4br6\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-wtmp\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-root\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e03dd829-a441-43bf-98bd-16395fd459ea-metrics-client-ca\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-sys\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-textfile\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.153850 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.153520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-tls\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.254705 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.254885 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-sys\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.254885 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-textfile\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.254885 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-tls\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.254885 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.254885 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4br6\" (UniqueName: \"kubernetes.io/projected/e03dd829-a441-43bf-98bd-16395fd459ea-kube-api-access-v4br6\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255166 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-wtmp\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255166 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-root\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255166 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.254973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e03dd829-a441-43bf-98bd-16395fd459ea-metrics-client-ca\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255351 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.255316    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-textfile\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255874 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.255390    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-sys\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255874 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.255611    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255874 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.255648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e03dd829-a441-43bf-98bd-16395fd459ea-metrics-client-ca\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255874 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.255778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-wtmp\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.255874 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.255819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e03dd829-a441-43bf-98bd-16395fd459ea-root\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.258716 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.258444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.258716 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.258660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e03dd829-a441-43bf-98bd-16395fd459ea-node-exporter-tls\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.263882 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.263806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4br6\" (UniqueName: \"kubernetes.io/projected/e03dd829-a441-43bf-98bd-16395fd459ea-kube-api-access-v4br6\") pod \"node-exporter-d4v5w\" (UID: \"e03dd829-a441-43bf-98bd-16395fd459ea\") " pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.400868 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.400831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4v5w"
Jul 28 00:47:38.413216 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:38.413183    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:43.398701 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:47:43.398664    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03dd829_a441_43bf_98bd_16395fd459ea.slice/crio-a62a7dfd75320345da9116e76e7647dad2d551e7f0228cc6c7d7c276b77ebb1c WatchSource:0}: Error finding container a62a7dfd75320345da9116e76e7647dad2d551e7f0228cc6c7d7c276b77ebb1c: Status 404 returned error can't find the container with id a62a7dfd75320345da9116e76e7647dad2d551e7f0228cc6c7d7c276b77ebb1c
Jul 28 00:47:43.430000 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.429959    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9" podUID="f0660dbe-4d29-4a5c-84b2-6e25b6af7603" containerName="registry" containerID="cri-o://659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7" gracePeriod=30
Jul 28 00:47:43.742699 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.742576    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:43.759797 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.759771    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0660dbe-4d29-4a5c-84b2-6e25b6af7603" containerID="659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7" exitCode=0
Jul 28 00:47:43.759971 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.759822    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9"
Jul 28 00:47:43.759971 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.759855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9" event={"ID":"f0660dbe-4d29-4a5c-84b2-6e25b6af7603","Type":"ContainerDied","Data":"659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7"}
Jul 28 00:47:43.759971 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.759910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbff6fc5-m2lf9" event={"ID":"f0660dbe-4d29-4a5c-84b2-6e25b6af7603","Type":"ContainerDied","Data":"e44bd818328a61cd39ef7c3582465f9c662d5a8cf48a5437349720256c2e94fd"}
Jul 28 00:47:43.759971 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.759965    2564 scope.go:117] "RemoveContainer" containerID="659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7"
Jul 28 00:47:43.761566 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.761539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-88wn4" event={"ID":"6b384be7-70d1-47a4-84be-fe3f3c8a2617","Type":"ContainerStarted","Data":"fcd6f999cf124f59876664bf7865ca6839f74f6e6a633ba066a199038efdd34e"}
Jul 28 00:47:43.762845 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.762808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4v5w" event={"ID":"e03dd829-a441-43bf-98bd-16395fd459ea","Type":"ContainerStarted","Data":"a62a7dfd75320345da9116e76e7647dad2d551e7f0228cc6c7d7c276b77ebb1c"}
Jul 28 00:47:43.771958 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.771912    2564 scope.go:117] "RemoveContainer" containerID="659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7"
Jul 28 00:47:43.772644 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:47:43.772600    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7\": container with ID starting with 659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7 not found: ID does not exist" containerID="659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7"
Jul 28 00:47:43.772747 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.772659    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7"} err="failed to get container status \"659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7\": rpc error: code = NotFound desc = could not find container \"659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7\": container with ID starting with 659bc54d992b97517803c9af0579400fa9f512c8172419f65fe3a8374db35fc7 not found: ID does not exist"
Jul 28 00:47:43.806608 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806525    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-trusted-ca\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806625    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-image-registry-private-configuration\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806645    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-installation-pull-secrets\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806662    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-bound-sa-token\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-ca-trust-extracted\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4q9q\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-kube-api-access-j4q9q\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.806768 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.806768    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-certificates\") pod \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\" (UID: \"f0660dbe-4d29-4a5c-84b2-6e25b6af7603\") "
Jul 28 00:47:43.807199 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.807165    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:47:43.807458 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.807429    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:47:43.809507 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.809473    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:47:43.809615 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.809525    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:47:43.809615 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.809557    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:47:43.809736 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.809718    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:47:43.809825 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.809716    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-kube-api-access-j4q9q" (OuterVolumeSpecName: "kube-api-access-j4q9q") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "kube-api-access-j4q9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:47:43.817394 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.817366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f0660dbe-4d29-4a5c-84b2-6e25b6af7603" (UID: "f0660dbe-4d29-4a5c-84b2-6e25b6af7603"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907626    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-certificates\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907663    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-trusted-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907679    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-image-registry-private-configuration\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907689    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-installation-pull-secrets\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907697    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-bound-sa-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907705    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-registry-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.907708 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907713    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-ca-trust-extracted\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:43.908045 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:43.907721    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4q9q\" (UniqueName: \"kubernetes.io/projected/f0660dbe-4d29-4a5c-84b2-6e25b6af7603-kube-api-access-j4q9q\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 00:47:44.088475 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:44.088408    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69dbff6fc5-m2lf9"]
Jul 28 00:47:44.098745 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:44.098690    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69dbff6fc5-m2lf9"]
Jul 28 00:47:44.769280 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:44.769075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:44.782172 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:44.782135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-88wn4"
Jul 28 00:47:44.787479 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:44.787433    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-88wn4" podStartSLOduration=7.830991708 podStartE2EDuration="28.787417846s" totalImagesPullingTime="20.956426138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:44.78547713 +0000 UTC m=+96.340980323" watchObservedRunningTime="2026-07-28 00:47:44.787417846 +0000 UTC m=+96.342921006"
Jul 28 00:47:44.945992 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:44.945956    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0660dbe-4d29-4a5c-84b2-6e25b6af7603" path="/var/lib/kubelet/pods/f0660dbe-4d29-4a5c-84b2-6e25b6af7603/volumes"
Jul 28 00:47:45.776282 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:45.776243    2564 generic.go:356] "Generic (PLEG): container finished" podID="e03dd829-a441-43bf-98bd-16395fd459ea" containerID="fb48ee5dbaf6eae91eed816c5eea39bce047a80822f7b4168b83f7b0e4b04597" exitCode=0
Jul 28 00:47:45.776746 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:45.776341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4v5w" event={"ID":"e03dd829-a441-43bf-98bd-16395fd459ea","Type":"ContainerDied","Data":"fb48ee5dbaf6eae91eed816c5eea39bce047a80822f7b4168b83f7b0e4b04597"}
Jul 28 00:47:46.668616 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:46.668584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d559cd4cf-d258t"
Jul 28 00:47:46.784067 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:46.784031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4v5w" event={"ID":"e03dd829-a441-43bf-98bd-16395fd459ea","Type":"ContainerStarted","Data":"b7e9c6f948d059bbc988452320994291a345a3ce97cff05f82de5897db9fcc33"}
Jul 28 00:47:46.784523 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:46.784089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4v5w" event={"ID":"e03dd829-a441-43bf-98bd-16395fd459ea","Type":"ContainerStarted","Data":"bfa8fbce97a6c05437a7ea5ddcbb3f5f691bb873bb92952a956cc9c47ed5fb66"}
Jul 28 00:47:46.803806 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:46.803757    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d4v5w" podStartSLOduration=8.040383047 podStartE2EDuration="9.803739903s" totalImagesPullingTime="1.649837295s" totalInitContainerRuntime="113.519561ms" isStatefulPod=true podCreationTimestamp="2026-07-28 00:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:46.803054312 +0000 UTC m=+98.358557495" watchObservedRunningTime="2026-07-28 00:47:46.803739903 +0000 UTC m=+98.359243063"
Jul 28 00:47:47.405406 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:47:47.405325    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z5mrk"
Jul 28 00:48:03.856684 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:03.856602    2564 generic.go:356] "Generic (PLEG): container finished" podID="4356f204-868b-4aa7-bfc2-bf271dbff21c" containerID="dadc2cdeba17a6862c1141937d3b4a3fd933d46d3eee1e5769924c4dcf6fd501" exitCode=0
Jul 28 00:48:03.856684 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:03.856673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9" event={"ID":"4356f204-868b-4aa7-bfc2-bf271dbff21c","Type":"ContainerDied","Data":"dadc2cdeba17a6862c1141937d3b4a3fd933d46d3eee1e5769924c4dcf6fd501"}
Jul 28 00:48:03.857159 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:03.857029    2564 scope.go:117] "RemoveContainer" containerID="dadc2cdeba17a6862c1141937d3b4a3fd933d46d3eee1e5769924c4dcf6fd501"
Jul 28 00:48:04.865672 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:04.865640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-65gc9" event={"ID":"4356f204-868b-4aa7-bfc2-bf271dbff21c","Type":"ContainerStarted","Data":"868c8daa3ba872bccde9bca1f9743ba3bb78e221223c2c53831f3d249ca20711"}
Jul 28 00:48:09.887566 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:09.887527    2564 generic.go:356] "Generic (PLEG): container finished" podID="91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4" containerID="3c16e8b90e0a6b733df1eabe46cfcb132f3854d2f07fef64ee38509d902e9cd6" exitCode=0
Jul 28 00:48:09.888046 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:09.887602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692" event={"ID":"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4","Type":"ContainerDied","Data":"3c16e8b90e0a6b733df1eabe46cfcb132f3854d2f07fef64ee38509d902e9cd6"}
Jul 28 00:48:09.888046 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:09.887950    2564 scope.go:117] "RemoveContainer" containerID="3c16e8b90e0a6b733df1eabe46cfcb132f3854d2f07fef64ee38509d902e9cd6"
Jul 28 00:48:10.893224 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:10.893192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc692" event={"ID":"91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4","Type":"ContainerStarted","Data":"b1b7f5b56dfe08c193f3f9e821805427331e82fb6b6a6279f7c2f853a98a772e"}
Jul 28 00:48:28.960511 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:28.960476    2564 generic.go:356] "Generic (PLEG): container finished" podID="6e6dacd7-8d50-4764-8313-e15b794713c3" containerID="5546af409280c0e7ef1eef5ea214c9b4087f27864dde2841d109792677a701b5" exitCode=0
Jul 28 00:48:28.960917 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:28.960549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kkl92" event={"ID":"6e6dacd7-8d50-4764-8313-e15b794713c3","Type":"ContainerDied","Data":"5546af409280c0e7ef1eef5ea214c9b4087f27864dde2841d109792677a701b5"}
Jul 28 00:48:28.960917 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:28.960903    2564 scope.go:117] "RemoveContainer" containerID="5546af409280c0e7ef1eef5ea214c9b4087f27864dde2841d109792677a701b5"
Jul 28 00:48:29.966588 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:48:29.966550    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kkl92" event={"ID":"6e6dacd7-8d50-4764-8313-e15b794713c3","Type":"ContainerStarted","Data":"09641dee859125365e6ad78a069b872fb630ef373182a3cf340bb5f5f40b34a4"}
Jul 28 00:51:08.840598 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:51:08.840567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:51:08.845410 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:51:08.845384    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:51:08.851879 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:51:08.851858    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 00:52:23.026253 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.026168    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"]
Jul 28 00:52:23.026743 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.026517    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0660dbe-4d29-4a5c-84b2-6e25b6af7603" containerName="registry"
Jul 28 00:52:23.026743 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.026531    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0660dbe-4d29-4a5c-84b2-6e25b6af7603" containerName="registry"
Jul 28 00:52:23.026743 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.026596    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0660dbe-4d29-4a5c-84b2-6e25b6af7603" containerName="registry"
Jul 28 00:52:23.028640 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.028624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.031120 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.031093    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s5btk\""
Jul 28 00:52:23.031295 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.031124    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 00:52:23.031407 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.031390    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 00:52:23.031479 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.031408    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 00:52:23.031724 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.031704    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 00:52:23.059564 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.059532    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"]
Jul 28 00:52:23.115378 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.115340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ab83145-2bda-46ac-a23e-157afe8580c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.115548 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.115386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab83145-2bda-46ac-a23e-157afe8580c0-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.115548 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.115408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ab83145-2bda-46ac-a23e-157afe8580c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.115548 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.115433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ab83145-2bda-46ac-a23e-157afe8580c0-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.115548 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.115457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmww\" (UniqueName: \"kubernetes.io/projected/4ab83145-2bda-46ac-a23e-157afe8580c0-kube-api-access-wdmww\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.215860 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.215818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab83145-2bda-46ac-a23e-157afe8580c0-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.215860 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.215863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ab83145-2bda-46ac-a23e-157afe8580c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.216147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.215903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ab83145-2bda-46ac-a23e-157afe8580c0-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.216147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.215963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmww\" (UniqueName: \"kubernetes.io/projected/4ab83145-2bda-46ac-a23e-157afe8580c0-kube-api-access-wdmww\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.216147 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.216026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ab83145-2bda-46ac-a23e-157afe8580c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.216304 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.216277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ab83145-2bda-46ac-a23e-157afe8580c0-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.218357 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.218328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab83145-2bda-46ac-a23e-157afe8580c0-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.219002 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.218978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ab83145-2bda-46ac-a23e-157afe8580c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.219106 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.219012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ab83145-2bda-46ac-a23e-157afe8580c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.224843 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.224815    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmww\" (UniqueName: \"kubernetes.io/projected/4ab83145-2bda-46ac-a23e-157afe8580c0-kube-api-access-wdmww\") pod \"opendatahub-operator-controller-manager-644f76477d-2k65g\" (UID: \"4ab83145-2bda-46ac-a23e-157afe8580c0\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.339381 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.339275    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:23.504762 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.504734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"]
Jul 28 00:52:23.507626 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:52:23.507596    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab83145_2bda_46ac_a23e_157afe8580c0.slice/crio-3542418bcd7e8489faaeddddee98d7296e0874d5c5ea3bddb78b1a9dd83cd08a WatchSource:0}: Error finding container 3542418bcd7e8489faaeddddee98d7296e0874d5c5ea3bddb78b1a9dd83cd08a: Status 404 returned error can't find the container with id 3542418bcd7e8489faaeddddee98d7296e0874d5c5ea3bddb78b1a9dd83cd08a
Jul 28 00:52:23.509389 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.509370    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 00:52:23.884236 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:23.884200    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g" event={"ID":"4ab83145-2bda-46ac-a23e-157afe8580c0","Type":"ContainerStarted","Data":"3542418bcd7e8489faaeddddee98d7296e0874d5c5ea3bddb78b1a9dd83cd08a"}
Jul 28 00:52:26.901514 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:26.901478    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g" event={"ID":"4ab83145-2bda-46ac-a23e-157afe8580c0","Type":"ContainerStarted","Data":"b1e96846eadfc2c9e082c76ab6eb25f76eb02a8f845bad9d04ec1ee315d30587"}
Jul 28 00:52:27.907707 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:27.907614    2564 generic.go:356] "Generic (PLEG): container finished" podID="4ab83145-2bda-46ac-a23e-157afe8580c0" containerID="b1e96846eadfc2c9e082c76ab6eb25f76eb02a8f845bad9d04ec1ee315d30587" exitCode=0
Jul 28 00:52:27.908089 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:27.907701    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g" event={"ID":"4ab83145-2bda-46ac-a23e-157afe8580c0","Type":"ContainerDied","Data":"b1e96846eadfc2c9e082c76ab6eb25f76eb02a8f845bad9d04ec1ee315d30587"}
Jul 28 00:52:28.914451 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:28.914415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g" event={"ID":"4ab83145-2bda-46ac-a23e-157afe8580c0","Type":"ContainerStarted","Data":"805a075c67c3fab1e7702c55c7658be55e15884d09fcc1c5101d3ac545fa77bd"}
Jul 28 00:52:28.914903 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:28.914536    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:28.941061 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:28.941006    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g" podStartSLOduration=2.6683685649999997 podStartE2EDuration="6.940990566s" totalImagesPullingTime="3.253575648s" totalInitContainerRuntime="1.019046353s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:28.938535359 +0000 UTC m=+380.494038515" watchObservedRunningTime="2026-07-28 00:52:28.940990566 +0000 UTC m=+380.496493723"
Jul 28 00:52:39.921778 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:39.921744    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-2k65g"
Jul 28 00:52:52.059058 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.059017    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"]
Jul 28 00:52:52.069020 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.068990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.072124 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.072060    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7jp5c\""
Jul 28 00:52:52.072657 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.072633    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 28 00:52:52.079234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.079211    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"]
Jul 28 00:52:52.157984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.157947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85bf38c3-8c69-4fd7-be47-85555a13d6f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.157984 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.157986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9lw\" (UniqueName: \"kubernetes.io/projected/85bf38c3-8c69-4fd7-be47-85555a13d6f5-kube-api-access-pq9lw\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.259328 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.259296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85bf38c3-8c69-4fd7-be47-85555a13d6f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.259328 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.259330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9lw\" (UniqueName: \"kubernetes.io/projected/85bf38c3-8c69-4fd7-be47-85555a13d6f5-kube-api-access-pq9lw\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.259545 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:52:52.259449    2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 00:52:52.259545 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:52:52.259519    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bf38c3-8c69-4fd7-be47-85555a13d6f5-cert podName:85bf38c3-8c69-4fd7-be47-85555a13d6f5 nodeName:}" failed. No retries permitted until 2026-07-28 00:52:52.759496793 +0000 UTC m=+404.314999940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/85bf38c3-8c69-4fd7-be47-85555a13d6f5-cert") pod "llmisvc-controller-manager-777c6bd7d5-hv8nt" (UID: "85bf38c3-8c69-4fd7-be47-85555a13d6f5") : secret "llmisvc-webhook-server-cert" not found
Jul 28 00:52:52.282038 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.282003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9lw\" (UniqueName: \"kubernetes.io/projected/85bf38c3-8c69-4fd7-be47-85555a13d6f5-kube-api-access-pq9lw\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.764083 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.764029    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85bf38c3-8c69-4fd7-be47-85555a13d6f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.772534 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.766522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85bf38c3-8c69-4fd7-be47-85555a13d6f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hv8nt\" (UID: \"85bf38c3-8c69-4fd7-be47-85555a13d6f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:52.983366 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:52.983325    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:53.130258 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.130075    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"]
Jul 28 00:52:53.133034 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:52:53.132998    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85bf38c3_8c69_4fd7_be47_85555a13d6f5.slice/crio-578488169c18ebc273c9bca94e8f07e2d13dd9ff064d5c4a7eb0a797a3b91a60 WatchSource:0}: Error finding container 578488169c18ebc273c9bca94e8f07e2d13dd9ff064d5c4a7eb0a797a3b91a60: Status 404 returned error can't find the container with id 578488169c18ebc273c9bca94e8f07e2d13dd9ff064d5c4a7eb0a797a3b91a60
Jul 28 00:52:53.274603 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.274568    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c44798b85-j295n"]
Jul 28 00:52:53.277491 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.277470    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.279771 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.279743    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 00:52:53.279875 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.279757    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 00:52:53.300602 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.300569    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c44798b85-j295n"]
Jul 28 00:52:53.369908 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.369873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkdq\" (UniqueName: \"kubernetes.io/projected/7bf5162b-e91f-4551-960b-a84ad77294a4-kube-api-access-dzkdq\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.370094 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.369924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf5162b-e91f-4551-960b-a84ad77294a4-tmp\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.370094 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.369965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf5162b-e91f-4551-960b-a84ad77294a4-tls-certs\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.471262 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.471220    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf5162b-e91f-4551-960b-a84ad77294a4-tmp\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.471262 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.471266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf5162b-e91f-4551-960b-a84ad77294a4-tls-certs\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.471490 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.471319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkdq\" (UniqueName: \"kubernetes.io/projected/7bf5162b-e91f-4551-960b-a84ad77294a4-kube-api-access-dzkdq\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.471490 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:52:53.471447    2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 00:52:53.471564 ip-10-0-131-111 kubenswrapper[2564]: E0728 00:52:53.471526    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf5162b-e91f-4551-960b-a84ad77294a4-tls-certs podName:7bf5162b-e91f-4551-960b-a84ad77294a4 nodeName:}" failed. No retries permitted until 2026-07-28 00:52:53.971503836 +0000 UTC m=+405.527006996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7bf5162b-e91f-4551-960b-a84ad77294a4-tls-certs") pod "kube-auth-proxy-c44798b85-j295n" (UID: "7bf5162b-e91f-4551-960b-a84ad77294a4") : secret "kube-auth-proxy-tls" not found
Jul 28 00:52:53.473552 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.473533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf5162b-e91f-4551-960b-a84ad77294a4-tmp\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.481739 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.481709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkdq\" (UniqueName: \"kubernetes.io/projected/7bf5162b-e91f-4551-960b-a84ad77294a4-kube-api-access-dzkdq\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.975979 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.975930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf5162b-e91f-4551-960b-a84ad77294a4-tls-certs\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:53.978456 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:53.978424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf5162b-e91f-4551-960b-a84ad77294a4-tls-certs\") pod \"kube-auth-proxy-c44798b85-j295n\" (UID: \"7bf5162b-e91f-4551-960b-a84ad77294a4\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:54.027521 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:54.027487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt" event={"ID":"85bf38c3-8c69-4fd7-be47-85555a13d6f5","Type":"ContainerStarted","Data":"578488169c18ebc273c9bca94e8f07e2d13dd9ff064d5c4a7eb0a797a3b91a60"}
Jul 28 00:52:54.187596 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:54.187555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n"
Jul 28 00:52:54.782088 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:54.782056    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c44798b85-j295n"]
Jul 28 00:52:54.783742 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:52:54.783696    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf5162b_e91f_4551_960b_a84ad77294a4.slice/crio-bfa96bcdfa3a13d7a70881ee5337d33991c7f23b98dea9f4cde58056f285a2db WatchSource:0}: Error finding container bfa96bcdfa3a13d7a70881ee5337d33991c7f23b98dea9f4cde58056f285a2db: Status 404 returned error can't find the container with id bfa96bcdfa3a13d7a70881ee5337d33991c7f23b98dea9f4cde58056f285a2db
Jul 28 00:52:55.033595 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.033555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n" event={"ID":"7bf5162b-e91f-4551-960b-a84ad77294a4","Type":"ContainerStarted","Data":"bfa96bcdfa3a13d7a70881ee5337d33991c7f23b98dea9f4cde58056f285a2db"}
Jul 28 00:52:55.085231 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.085188    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"]
Jul 28 00:52:55.088858 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.088833    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.091142 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.091122    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 00:52:55.091257 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.091156    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 00:52:55.091327 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.091258    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 00:52:55.091327 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.091275    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n8fg7\""
Jul 28 00:52:55.091439 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.091379    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 00:52:55.103413 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.103386    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"]
Jul 28 00:52:55.188735 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.188693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.188735 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.188738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.189234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.188773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.189234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.188799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhgj\" (UniqueName: \"kubernetes.io/projected/26d9efc1-24d6-4435-ad24-cb5a1712c834-kube-api-access-tzhgj\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.189234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.188907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.189234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.188981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d9efc1-24d6-4435-ad24-cb5a1712c834-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.189234 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.189002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290383 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290383 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d9efc1-24d6-4435-ad24-cb5a1712c834-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290383 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290653 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290653 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290653 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.290653 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.290496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhgj\" (UniqueName: \"kubernetes.io/projected/26d9efc1-24d6-4435-ad24-cb5a1712c834-kube-api-access-tzhgj\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.291646 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.291617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.293878 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.293825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.293878 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.293831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.294079 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.293976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.294351 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.294330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d9efc1-24d6-4435-ad24-cb5a1712c834-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.305777 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.305742    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhgj\" (UniqueName: \"kubernetes.io/projected/26d9efc1-24d6-4435-ad24-cb5a1712c834-kube-api-access-tzhgj\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.309963 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.309916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d9efc1-24d6-4435-ad24-cb5a1712c834-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cck8t\" (UID: \"26d9efc1-24d6-4435-ad24-cb5a1712c834\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.399127 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.399089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:52:55.574191 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:55.574160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"]
Jul 28 00:52:56.135428 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:52:56.135384    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d9efc1_24d6_4435_ad24_cb5a1712c834.slice/crio-235ac0406f5b7fe8c38dcfa6b9861a708f9bf11617d87e5176dbd3d13d81032f WatchSource:0}: Error finding container 235ac0406f5b7fe8c38dcfa6b9861a708f9bf11617d87e5176dbd3d13d81032f: Status 404 returned error can't find the container with id 235ac0406f5b7fe8c38dcfa6b9861a708f9bf11617d87e5176dbd3d13d81032f
Jul 28 00:52:57.046080 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:57.046043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt" event={"ID":"85bf38c3-8c69-4fd7-be47-85555a13d6f5","Type":"ContainerStarted","Data":"58bc8650d03171500cea2480c34e35108156a45fcf1a70f9caa537171bf66ab9"}
Jul 28 00:52:57.046562 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:57.046162    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:52:57.047308 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:57.047283    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t" event={"ID":"26d9efc1-24d6-4435-ad24-cb5a1712c834","Type":"ContainerStarted","Data":"235ac0406f5b7fe8c38dcfa6b9861a708f9bf11617d87e5176dbd3d13d81032f"}
Jul 28 00:52:57.062522 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:57.062463    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt" podStartSLOduration=2.010729163 podStartE2EDuration="5.062441912s" totalImagesPullingTime="3.051712749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:57.06061674 +0000 UTC m=+408.616119925" watchObservedRunningTime="2026-07-28 00:52:57.062441912 +0000 UTC m=+408.617945076"
Jul 28 00:52:58.056038 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:58.055996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n" event={"ID":"7bf5162b-e91f-4551-960b-a84ad77294a4","Type":"ContainerStarted","Data":"0247527be456320a9835f6d6b5d349efedc5f814c4853fc9b1a6bb2c05d57d66"}
Jul 28 00:52:58.076087 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:58.076032    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c44798b85-j295n" podStartSLOduration=2.775430792 podStartE2EDuration="5.076014936s" totalImagesPullingTime="2.300584144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:58.073984739 +0000 UTC m=+409.629487932" watchObservedRunningTime="2026-07-28 00:52:58.076014936 +0000 UTC m=+409.631518095"
Jul 28 00:52:59.248559 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:59.248515    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 00:52:59.248905 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:52:59.248590    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 00:53:00.068512 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:00.068477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t" event={"ID":"26d9efc1-24d6-4435-ad24-cb5a1712c834","Type":"ContainerStarted","Data":"8ad527dd1772f2bc40b5c89e0671ab210fd7089fd73a097df100a0843e15e182"}
Jul 28 00:53:00.068747 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:00.068724    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:53:00.070592 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:00.070542    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-cck8t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 00:53:00.070724 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:00.070598    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t" podUID="26d9efc1-24d6-4435-ad24-cb5a1712c834" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 00:53:00.091925 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:00.091849    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t" podStartSLOduration=1.98153346 podStartE2EDuration="5.091828627s" totalImagesPullingTime="3.110295167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:53:00.09075691 +0000 UTC m=+411.646260070" watchObservedRunningTime="2026-07-28 00:53:00.091828627 +0000 UTC m=+411.647331789"
Jul 28 00:53:01.074178 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:01.074143    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cck8t"
Jul 28 00:53:28.058398 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:53:28.058364    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hv8nt"
Jul 28 00:54:41.966747 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:41.966711    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"]
Jul 28 00:54:41.970201 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:41.970180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:54:41.977850 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:41.977823    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dv5qf\""
Jul 28 00:54:41.978018 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:41.977827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 00:54:41.981141 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:41.981116    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 00:54:41.989602 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:41.989558    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"]
Jul 28 00:54:42.020156 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:42.020113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4tj\" (UniqueName: \"kubernetes.io/projected/e003d4a9-9e4c-4c72-8e29-748e2865e74b-kube-api-access-tl4tj\") pod \"limitador-operator-controller-manager-859d7b78bd-mv8gl\" (UID: \"e003d4a9-9e4c-4c72-8e29-748e2865e74b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:54:42.121382 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:42.121337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4tj\" (UniqueName: \"kubernetes.io/projected/e003d4a9-9e4c-4c72-8e29-748e2865e74b-kube-api-access-tl4tj\") pod \"limitador-operator-controller-manager-859d7b78bd-mv8gl\" (UID: \"e003d4a9-9e4c-4c72-8e29-748e2865e74b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:54:42.153774 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:42.153737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4tj\" (UniqueName: \"kubernetes.io/projected/e003d4a9-9e4c-4c72-8e29-748e2865e74b-kube-api-access-tl4tj\") pod \"limitador-operator-controller-manager-859d7b78bd-mv8gl\" (UID: \"e003d4a9-9e4c-4c72-8e29-748e2865e74b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:54:42.281518 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:42.281419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:54:42.440090 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:42.440065    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"]
Jul 28 00:54:42.442751 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:54:42.442723    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode003d4a9_9e4c_4c72_8e29_748e2865e74b.slice/crio-a4cdf71bbf54e46ac75d90e3ad413f3638a0472368010b8c1f28d477b7928769 WatchSource:0}: Error finding container a4cdf71bbf54e46ac75d90e3ad413f3638a0472368010b8c1f28d477b7928769: Status 404 returned error can't find the container with id a4cdf71bbf54e46ac75d90e3ad413f3638a0472368010b8c1f28d477b7928769
Jul 28 00:54:42.529180 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:42.529146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl" event={"ID":"e003d4a9-9e4c-4c72-8e29-748e2865e74b","Type":"ContainerStarted","Data":"a4cdf71bbf54e46ac75d90e3ad413f3638a0472368010b8c1f28d477b7928769"}
Jul 28 00:54:44.539911 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:44.539872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl" event={"ID":"e003d4a9-9e4c-4c72-8e29-748e2865e74b","Type":"ContainerStarted","Data":"edc94eebec977342dbbf1091dcf1fd4b0b8b8a6bf4a28e7773288714f9e92b5a"}
Jul 28 00:54:44.540316 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:44.539953    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:54:44.572440 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:44.572384    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl" podStartSLOduration=2.073959498 podStartE2EDuration="3.572367467s" totalImagesPullingTime="1.498407969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:44.570960173 +0000 UTC m=+516.126463359" watchObservedRunningTime="2026-07-28 00:54:44.572367467 +0000 UTC m=+516.127870627"
Jul 28 00:54:55.546358 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:54:55.546319    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mv8gl"
Jul 28 00:55:19.730340 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.730255    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:55:19.736411 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.736385    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:19.738663 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.738627    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 00:55:19.738663 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.738627    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qx2mk\""
Jul 28 00:55:19.746023 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.745988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlsg\" (UniqueName: \"kubernetes.io/projected/8e5418a1-6f76-4259-a5c8-0ae65de93408-kube-api-access-4rlsg\") pod \"limitador-limitador-54d6d988d5-hrvn7\" (UID: \"8e5418a1-6f76-4259-a5c8-0ae65de93408\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:19.746235 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.746217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e5418a1-6f76-4259-a5c8-0ae65de93408-config-file\") pod \"limitador-limitador-54d6d988d5-hrvn7\" (UID: \"8e5418a1-6f76-4259-a5c8-0ae65de93408\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:19.748888 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.748849    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:55:19.767036 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.767000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:55:19.846724 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.846686    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlsg\" (UniqueName: \"kubernetes.io/projected/8e5418a1-6f76-4259-a5c8-0ae65de93408-kube-api-access-4rlsg\") pod \"limitador-limitador-54d6d988d5-hrvn7\" (UID: \"8e5418a1-6f76-4259-a5c8-0ae65de93408\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:19.846724 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.846730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e5418a1-6f76-4259-a5c8-0ae65de93408-config-file\") pod \"limitador-limitador-54d6d988d5-hrvn7\" (UID: \"8e5418a1-6f76-4259-a5c8-0ae65de93408\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:19.847387 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.847368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e5418a1-6f76-4259-a5c8-0ae65de93408-config-file\") pod \"limitador-limitador-54d6d988d5-hrvn7\" (UID: \"8e5418a1-6f76-4259-a5c8-0ae65de93408\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:19.854943 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:19.854905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlsg\" (UniqueName: \"kubernetes.io/projected/8e5418a1-6f76-4259-a5c8-0ae65de93408-kube-api-access-4rlsg\") pod \"limitador-limitador-54d6d988d5-hrvn7\" (UID: \"8e5418a1-6f76-4259-a5c8-0ae65de93408\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:20.051355 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:20.051263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:20.189683 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:20.189645    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:55:20.193092 ip-10-0-131-111 kubenswrapper[2564]: W0728 00:55:20.193055    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5418a1_6f76_4259_a5c8_0ae65de93408.slice/crio-e93d6985ee35f229e33d4e658451b66f9c20889adad3a7e2701ffff79c18c0e1 WatchSource:0}: Error finding container e93d6985ee35f229e33d4e658451b66f9c20889adad3a7e2701ffff79c18c0e1: Status 404 returned error can't find the container with id e93d6985ee35f229e33d4e658451b66f9c20889adad3a7e2701ffff79c18c0e1
Jul 28 00:55:20.707974 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:20.707863    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7" event={"ID":"8e5418a1-6f76-4259-a5c8-0ae65de93408","Type":"ContainerStarted","Data":"e93d6985ee35f229e33d4e658451b66f9c20889adad3a7e2701ffff79c18c0e1"}
Jul 28 00:55:22.719177 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:22.719145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7" event={"ID":"8e5418a1-6f76-4259-a5c8-0ae65de93408","Type":"ContainerStarted","Data":"faf5f637e296852abc7db9667c5b6d7eeb86f211615db99f2bcf40b0991a079b"}
Jul 28 00:55:22.719565 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:22.719268    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:33.724623 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:33.724592    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"
Jul 28 00:55:33.741094 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:55:33.741040    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-hrvn7" podStartSLOduration=12.3665441 podStartE2EDuration="14.741023057s" totalImagesPullingTime="2.374478957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:22.73371826 +0000 UTC m=+554.289221441" watchObservedRunningTime="2026-07-28 00:55:33.741023057 +0000 UTC m=+565.296526217"
Jul 28 00:56:08.872959 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:56:08.872913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:56:08.874761 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:56:08.874739    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 00:56:31.844102 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:56:31.844014    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:56:54.199509 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:56:54.199477    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:57:07.009616 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:57:07.009568    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:57:17.998577 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:57:17.998532    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:57:23.992428 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:57:23.992393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:57:31.709166 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:57:31.709131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:57:39.697449 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:57:39.697413    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:59:08.405289 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:59:08.405254    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:59:17.801743 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:59:17.801683    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 00:59:28.003733 ip-10-0-131-111 kubenswrapper[2564]: I0728 00:59:28.003690    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:00:00.140147 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.140110    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"]
Jul 28 01:00:00.143402 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.143380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"
Jul 28 01:00:00.145516 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.145492    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:00:00.150193 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.150158    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"]
Jul 28 01:00:00.155136 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.155107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"
Jul 28 01:00:00.299259 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.299225    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"]
Jul 28 01:00:00.302351 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:00:00.302314    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edcc857_1946_420c_bc26_e3f9258a635c.slice/crio-bfad5a59db70dfad2ef879d603eb693245870d46a3a5cc3bd339584ae1171fcd WatchSource:0}: Error finding container bfad5a59db70dfad2ef879d603eb693245870d46a3a5cc3bd339584ae1171fcd: Status 404 returned error can't find the container with id bfad5a59db70dfad2ef879d603eb693245870d46a3a5cc3bd339584ae1171fcd
Jul 28 01:00:00.304353 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.304337    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:00:00.966992 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:00.966954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf" event={"ID":"3edcc857-1946-420c-bc26-e3f9258a635c","Type":"ContainerStarted","Data":"bfad5a59db70dfad2ef879d603eb693245870d46a3a5cc3bd339584ae1171fcd"}
Jul 28 01:00:03.983386 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:03.983350    2564 generic.go:356] "Generic (PLEG): container finished" podID="3edcc857-1946-420c-bc26-e3f9258a635c" containerID="e7be0a8a779e7bc18f164b2523eb73d1ccea74c5d91d1d41a2abb46e3b7cf857" exitCode=0
Jul 28 01:00:03.983776 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:03.983398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf" event={"ID":"3edcc857-1946-420c-bc26-e3f9258a635c","Type":"ContainerDied","Data":"e7be0a8a779e7bc18f164b2523eb73d1ccea74c5d91d1d41a2abb46e3b7cf857"}
Jul 28 01:00:05.128420 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:05.128394    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"
Jul 28 01:00:05.998511 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:05.998477    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf"
Jul 28 01:00:05.998661 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:05.998476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753340-mbzqf" event={"ID":"3edcc857-1946-420c-bc26-e3f9258a635c","Type":"ContainerDied","Data":"bfad5a59db70dfad2ef879d603eb693245870d46a3a5cc3bd339584ae1171fcd"}
Jul 28 01:00:05.998661 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:00:05.998623    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfad5a59db70dfad2ef879d603eb693245870d46a3a5cc3bd339584ae1171fcd"
Jul 28 01:01:08.907901 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:08.907827    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:01:08.907901 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:08.907827    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:01:09.098975 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:09.098927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:01:19.207027 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:19.206988    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:01:30.809809 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:30.809768    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:01:48.199538 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:48.199498    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:01:57.897320 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:01:57.897282    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:02:06.123361 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:02:06.123323    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:02:14.806854 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:02:14.806819    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:02:25.412438 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:02:25.412393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:02:35.999922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:02:35.999837    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:02:46.793195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:02:46.793157    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:02:56.489334 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:02:56.489292    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:03:19.107639 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:03:19.107604    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:03:28.399283 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:03:28.399245    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:03:52.204250 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:03:52.204207    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:04:02.197875 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:04:02.197783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:04:11.603540 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:04:11.603498    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:04:28.807200 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:04:28.807161    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:04:38.897764 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:04:38.897725    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:04:54.899157 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:04:54.899120    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:06:08.935499 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:06:08.935469    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:06:08.936756 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:06:08.936732    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:06:57.392991 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:06:57.392956    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:07:07.804336 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:07:07.804254    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:07:22.507095 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:07:22.507056    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:07:33.902149 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:07:33.902104    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:07:43.899519 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:07:43.899481    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:07:52.928246 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:07:52.928209    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:08:03.500033 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:08:03.499993    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:08:15.302891 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:08:15.302854    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:08:23.019954 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:08:23.019906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:08:34.091368 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:08:34.091291    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:08:45.472162 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:08:45.472127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:08:54.515325 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:08:54.515285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:09:11.493648 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:09:11.493613    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:09:20.396055 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:09:20.396017    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:09:33.504471 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:09:33.504428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:10:04.004359 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:10:04.004278    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:10:17.500542 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:10:17.500497    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:10:26.104081 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:10:26.104043    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:10:37.402299 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:10:37.402262    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:10:46.603475 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:10:46.603436    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:10:55.403863 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:10:55.403825    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:11:03.391828 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:03.391789    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:11:08.964607 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:08.964580    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:11:08.965879 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:08.965857    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:11:13.606795 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:13.606756    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:11:23.091316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:23.091280    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:11:36.400585 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:36.400499    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:11:45.703340 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:45.703296    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:11:55.602572 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:11:55.602534    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:12:05.096873 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:12:05.096837    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:12:17.400437 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:12:17.400398    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:12:25.303536 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:12:25.303495    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:12:37.601266 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:12:37.601228    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:12:49.906831 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:12:49.906792    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:13:03.395371 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:03.395287    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:13:16.101228 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:16.101194    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:13:28.304014 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:28.303974    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:13:40.603226 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:40.603189    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:13:51.926047 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:51.926003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:13:55.600615 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.600573    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"]
Jul 28 01:13:55.601123 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.601107    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edcc857-1946-420c-bc26-e3f9258a635c" containerName="cleanup"
Jul 28 01:13:55.601173 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.601126    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edcc857-1946-420c-bc26-e3f9258a635c" containerName="cleanup"
Jul 28 01:13:55.601232 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.601220    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3edcc857-1946-420c-bc26-e3f9258a635c" containerName="cleanup"
Jul 28 01:13:55.604562 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.604541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.610631 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.610602    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:13:55.610800 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.610770    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:13:55.611611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.611508    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:13:55.611611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.611524    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1dcfab82-serving-cert\""
Jul 28 01:13:55.639810 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.639776    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"]
Jul 28 01:13:55.720415 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.720376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23387723-d4cf-4167-9873-c864b751948c-openshift-service-ca\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.720590 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.720442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.720590 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.720472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4w2\" (UniqueName: \"kubernetes.io/projected/23387723-d4cf-4167-9873-c864b751948c-kube-api-access-nx4w2\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.821313 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.821270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23387723-d4cf-4167-9873-c864b751948c-openshift-service-ca\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.821514 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.821334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.821514 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.821364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4w2\" (UniqueName: \"kubernetes.io/projected/23387723-d4cf-4167-9873-c864b751948c-kube-api-access-nx4w2\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.821514 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:13:55.821500    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-serving-cert: secret "maas-api-e2e-ait-1dcfab82-serving-cert" not found
Jul 28 01:13:55.821636 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:13:55.821570    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls podName:23387723-d4cf-4167-9873-c864b751948c nodeName:}" failed. No retries permitted until 2026-07-28 01:13:56.321552795 +0000 UTC m=+1667.877055933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls") pod "maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" (UID: "23387723-d4cf-4167-9873-c864b751948c") : secret "maas-api-e2e-ait-1dcfab82-serving-cert" not found
Jul 28 01:13:55.822088 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.822067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23387723-d4cf-4167-9873-c864b751948c-openshift-service-ca\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:55.831296 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:55.831262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4w2\" (UniqueName: \"kubernetes.io/projected/23387723-d4cf-4167-9873-c864b751948c-kube-api-access-nx4w2\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:56.326848 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:56.326807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:56.329508 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:56.329480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls\") pod \"maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:56.515334 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:56.515289    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:56.653438 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:56.653405    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"]
Jul 28 01:13:56.656045 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:13:56.656009    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23387723_d4cf_4167_9873_c864b751948c.slice/crio-8b3bef99f0e7965cdcbd3c087d416edfd784086071e49ed614c34ba30b73f1a1 WatchSource:0}: Error finding container 8b3bef99f0e7965cdcbd3c087d416edfd784086071e49ed614c34ba30b73f1a1: Status 404 returned error can't find the container with id 8b3bef99f0e7965cdcbd3c087d416edfd784086071e49ed614c34ba30b73f1a1
Jul 28 01:13:56.657392 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:56.657367    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:13:56.809861 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:56.809823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" event={"ID":"23387723-d4cf-4167-9873-c864b751948c","Type":"ContainerStarted","Data":"8b3bef99f0e7965cdcbd3c087d416edfd784086071e49ed614c34ba30b73f1a1"}
Jul 28 01:13:59.831121 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:59.831071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" event={"ID":"23387723-d4cf-4167-9873-c864b751948c","Type":"ContainerStarted","Data":"9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32"}
Jul 28 01:13:59.831527 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:59.831201    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:13:59.847698 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:13:59.847651    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" podStartSLOduration=2.532654992 podStartE2EDuration="4.847635736s" totalImagesPullingTime="2.314980744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:13:59.84642685 +0000 UTC m=+1671.401930010" watchObservedRunningTime="2026-07-28 01:13:59.847635736 +0000 UTC m=+1671.403138895"
Jul 28 01:14:02.800009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:02.799968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:14:05.841982 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:05.841948    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:14:20.740209 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.740171    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"]
Jul 28 01:14:20.743505 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.743486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:20.746338 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.746317    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:14:20.752248 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.752222    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"]
Jul 28 01:14:20.821005 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.820971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29d7413-c135-40b0-83cf-30c0c4905a25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm\" (UID: \"d29d7413-c135-40b0-83cf-30c0c4905a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:20.922337 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.922297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29d7413-c135-40b0-83cf-30c0c4905a25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm\" (UID: \"d29d7413-c135-40b0-83cf-30c0c4905a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:20.922915 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:20.922892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29d7413-c135-40b0-83cf-30c0c4905a25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm\" (UID: \"d29d7413-c135-40b0-83cf-30c0c4905a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:21.054958 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:21.054863    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:21.192555 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:21.192530    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"]
Jul 28 01:14:21.194883 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:14:21.194854    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29d7413_c135_40b0_83cf_30c0c4905a25.slice/crio-69a1413a9f2a151255cbac47a697b81de8caef84625ab40fbcbcb85b3660a732 WatchSource:0}: Error finding container 69a1413a9f2a151255cbac47a697b81de8caef84625ab40fbcbcb85b3660a732: Status 404 returned error can't find the container with id 69a1413a9f2a151255cbac47a697b81de8caef84625ab40fbcbcb85b3660a732
Jul 28 01:14:21.936651 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:21.936617    2564 generic.go:356] "Generic (PLEG): container finished" podID="d29d7413-c135-40b0-83cf-30c0c4905a25" containerID="a2533e6d58ad2c7d9f9c728c7d182be8ffc83e0121e1ca68cf1e42dc45386dd8" exitCode=0
Jul 28 01:14:21.937036 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:21.936671    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm" event={"ID":"d29d7413-c135-40b0-83cf-30c0c4905a25","Type":"ContainerDied","Data":"a2533e6d58ad2c7d9f9c728c7d182be8ffc83e0121e1ca68cf1e42dc45386dd8"}
Jul 28 01:14:21.937036 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:21.936724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm" event={"ID":"d29d7413-c135-40b0-83cf-30c0c4905a25","Type":"ContainerStarted","Data":"69a1413a9f2a151255cbac47a697b81de8caef84625ab40fbcbcb85b3660a732"}
Jul 28 01:14:23.083186 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.083164    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:23.140755 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.140720    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29d7413-c135-40b0-83cf-30c0c4905a25-maas-api-service-ca\") pod \"d29d7413-c135-40b0-83cf-30c0c4905a25\" (UID: \"d29d7413-c135-40b0-83cf-30c0c4905a25\") "
Jul 28 01:14:23.141124 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.141100    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29d7413-c135-40b0-83cf-30c0c4905a25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d29d7413-c135-40b0-83cf-30c0c4905a25" (UID: "d29d7413-c135-40b0-83cf-30c0c4905a25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:14:23.242321 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.242233    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29d7413-c135-40b0-83cf-30c0c4905a25-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:14:23.948970 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.948929    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"
Jul 28 01:14:23.949140 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.948965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm" event={"ID":"d29d7413-c135-40b0-83cf-30c0c4905a25","Type":"ContainerDied","Data":"69a1413a9f2a151255cbac47a697b81de8caef84625ab40fbcbcb85b3660a732"}
Jul 28 01:14:23.949140 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:23.949021    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a1413a9f2a151255cbac47a697b81de8caef84625ab40fbcbcb85b3660a732"
Jul 28 01:14:31.304683 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.303511    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"]
Jul 28 01:14:31.304683 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.304185    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" podUID="23387723-d4cf-4167-9873-c864b751948c" containerName="maas-api" containerID="cri-o://9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32" gracePeriod=30
Jul 28 01:14:31.581162 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.581133    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:14:31.717924 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.717884    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx4w2\" (UniqueName: \"kubernetes.io/projected/23387723-d4cf-4167-9873-c864b751948c-kube-api-access-nx4w2\") pod \"23387723-d4cf-4167-9873-c864b751948c\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") "
Jul 28 01:14:31.717924 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.717923    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23387723-d4cf-4167-9873-c864b751948c-openshift-service-ca\") pod \"23387723-d4cf-4167-9873-c864b751948c\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") "
Jul 28 01:14:31.718211 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.718025    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls\") pod \"23387723-d4cf-4167-9873-c864b751948c\" (UID: \"23387723-d4cf-4167-9873-c864b751948c\") "
Jul 28 01:14:31.718361 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.718333    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23387723-d4cf-4167-9873-c864b751948c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23387723-d4cf-4167-9873-c864b751948c" (UID: "23387723-d4cf-4167-9873-c864b751948c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:14:31.720220 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.720193    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23387723-d4cf-4167-9873-c864b751948c-kube-api-access-nx4w2" (OuterVolumeSpecName: "kube-api-access-nx4w2") pod "23387723-d4cf-4167-9873-c864b751948c" (UID: "23387723-d4cf-4167-9873-c864b751948c"). InnerVolumeSpecName "kube-api-access-nx4w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:14:31.720284 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.720200    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23387723-d4cf-4167-9873-c864b751948c" (UID: "23387723-d4cf-4167-9873-c864b751948c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:14:31.819534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.819446    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4w2\" (UniqueName: \"kubernetes.io/projected/23387723-d4cf-4167-9873-c864b751948c-kube-api-access-nx4w2\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:14:31.819534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.819478    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23387723-d4cf-4167-9873-c864b751948c-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:14:31.819534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.819488    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23387723-d4cf-4167-9873-c864b751948c-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:14:31.985749 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.985705    2564 generic.go:356] "Generic (PLEG): container finished" podID="23387723-d4cf-4167-9873-c864b751948c" containerID="9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32" exitCode=0
Jul 28 01:14:31.985922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.985780    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"
Jul 28 01:14:31.985922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.985797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" event={"ID":"23387723-d4cf-4167-9873-c864b751948c","Type":"ContainerDied","Data":"9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32"}
Jul 28 01:14:31.985922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.985852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk" event={"ID":"23387723-d4cf-4167-9873-c864b751948c","Type":"ContainerDied","Data":"8b3bef99f0e7965cdcbd3c087d416edfd784086071e49ed614c34ba30b73f1a1"}
Jul 28 01:14:31.985922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.985880    2564 scope.go:117] "RemoveContainer" containerID="9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32"
Jul 28 01:14:31.995795 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.995777    2564 scope.go:117] "RemoveContainer" containerID="9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32"
Jul 28 01:14:31.996063 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:14:31.996043    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32\": container with ID starting with 9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32 not found: ID does not exist" containerID="9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32"
Jul 28 01:14:31.996136 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:31.996072    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32"} err="failed to get container status \"9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32\": rpc error: code = NotFound desc = could not find container \"9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32\": container with ID starting with 9060d2c9e0b049e319233e369755dca289fb16b9b61f494273e4b51eb5abbf32 not found: ID does not exist"
Jul 28 01:14:32.006391 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:32.006367    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"]
Jul 28 01:14:32.012371 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:32.012347    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcfab82-6d8477566f-kvdwk"]
Jul 28 01:14:32.944836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:32.944804    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23387723-d4cf-4167-9873-c864b751948c" path="/var/lib/kubelet/pods/23387723-d4cf-4167-9873-c864b751948c/volumes"
Jul 28 01:14:34.297300 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:34.297265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:14:35.816268 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:35.816227    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"]
Jul 28 01:14:35.820689 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:35.820661    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcfab82-bs4zm"]
Jul 28 01:14:36.946341 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.946310    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29d7413-c135-40b0-83cf-30c0c4905a25" path="/var/lib/kubelet/pods/d29d7413-c135-40b0-83cf-30c0c4905a25/volumes"
Jul 28 01:14:36.961995 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.961959    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"]
Jul 28 01:14:36.962431 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.962406    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29d7413-c135-40b0-83cf-30c0c4905a25" containerName="revoke-keys"
Jul 28 01:14:36.962431 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.962432    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29d7413-c135-40b0-83cf-30c0c4905a25" containerName="revoke-keys"
Jul 28 01:14:36.962633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.962457    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23387723-d4cf-4167-9873-c864b751948c" containerName="maas-api"
Jul 28 01:14:36.962633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.962466    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23387723-d4cf-4167-9873-c864b751948c" containerName="maas-api"
Jul 28 01:14:36.962633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.962547    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23387723-d4cf-4167-9873-c864b751948c" containerName="maas-api"
Jul 28 01:14:36.962633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.962559    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29d7413-c135-40b0-83cf-30c0c4905a25" containerName="revoke-keys"
Jul 28 01:14:36.967490 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.967469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:36.969870 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.969847    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d990b63a-gw-openshift-default-dockercfg-j79z7\""
Jul 28 01:14:36.980449 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:36.980420    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"]
Jul 28 01:14:37.064577 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-socket\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99168c-2c8b-4a38-a135-a2610532faac-istio-podinfo\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-certs\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-data\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-istio-token\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064874 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-credential-socket\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064874 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99168c-2c8b-4a38-a135-a2610532faac-istiod-ca-cert\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064874 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-envoy\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.064999 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.064909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsck\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-kube-api-access-7zsck\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.165980 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.165943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-certs\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166166 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-data\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166166 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-istio-token\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166287 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-credential-socket\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166287 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99168c-2c8b-4a38-a135-a2610532faac-istiod-ca-cert\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166287 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-envoy\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166442 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsck\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-kube-api-access-7zsck\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166442 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-socket\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166442 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99168c-2c8b-4a38-a135-a2610532faac-istio-podinfo\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166643 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-certs\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166847 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-credential-socket\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166847 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-data\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.166989 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.166902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-socket\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.167059 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.167003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99168c-2c8b-4a38-a135-a2610532faac-istiod-ca-cert\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.168797 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.168770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-envoy\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.169051 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.169035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99168c-2c8b-4a38-a135-a2610532faac-istio-podinfo\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.177174 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.177154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-istio-token\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.177291 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.177274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsck\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-kube-api-access-7zsck\") pod \"e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") " pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.279142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.279057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:37.424005 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:37.423977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"]
Jul 28 01:14:37.426169 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:14:37.426138    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f99168c_2c8b_4a38_a135_a2610532faac.slice/crio-200410ae9dc4fc0c446ae80f9a50fed1e875a6712ab14e858ae05d606fb9df49 WatchSource:0}: Error finding container 200410ae9dc4fc0c446ae80f9a50fed1e875a6712ab14e858ae05d606fb9df49: Status 404 returned error can't find the container with id 200410ae9dc4fc0c446ae80f9a50fed1e875a6712ab14e858ae05d606fb9df49
Jul 28 01:14:38.016139 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:38.016100    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" event={"ID":"6f99168c-2c8b-4a38-a135-a2610532faac","Type":"ContainerStarted","Data":"200410ae9dc4fc0c446ae80f9a50fed1e875a6712ab14e858ae05d606fb9df49"}
Jul 28 01:14:40.866142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:40.866105    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:14:40.866429 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:40.866176    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:14:40.866429 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:40.866206    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:14:41.033020 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:41.032980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" event={"ID":"6f99168c-2c8b-4a38-a135-a2610532faac","Type":"ContainerStarted","Data":"c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"}
Jul 28 01:14:41.053009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:41.052930    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" podStartSLOduration=1.615430666 podStartE2EDuration="5.052915586s" totalImagesPullingTime="3.43748492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:14:41.050631831 +0000 UTC m=+1712.606135014" watchObservedRunningTime="2026-07-28 01:14:41.052915586 +0000 UTC m=+1712.608418744"
Jul 28 01:14:41.279423 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:41.279382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:41.284129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:41.284101    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:42.037800 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.037740    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:42.038762 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.038740    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:42.557891 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.557853    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"]
Jul 28 01:14:42.560930 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.560905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.563377 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.563348    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:14:42.564250 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.564225    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:14:42.564360 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.564236    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d990b63a-serving-cert\""
Jul 28 01:14:42.564425 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.564397    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:14:42.570065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.570040    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"]
Jul 28 01:14:42.715499 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.715454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcn8\" (UniqueName: \"kubernetes.io/projected/58b571a4-4729-457f-af4a-359f7522a8ec-kube-api-access-twcn8\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.715666 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.715526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b571a4-4729-457f-af4a-359f7522a8ec-maas-api-tls\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.715666 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.715578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b571a4-4729-457f-af4a-359f7522a8ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.816303 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.816201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcn8\" (UniqueName: \"kubernetes.io/projected/58b571a4-4729-457f-af4a-359f7522a8ec-kube-api-access-twcn8\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.816303 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.816269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b571a4-4729-457f-af4a-359f7522a8ec-maas-api-tls\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.816514 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.816319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b571a4-4729-457f-af4a-359f7522a8ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.817029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.817004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b571a4-4729-457f-af4a-359f7522a8ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.818705 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.818685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b571a4-4729-457f-af4a-359f7522a8ec-maas-api-tls\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.823682 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.823658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcn8\" (UniqueName: \"kubernetes.io/projected/58b571a4-4729-457f-af4a-359f7522a8ec-kube-api-access-twcn8\") pod \"maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:42.874555 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:42.874519    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:43.011466 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:43.011441    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"]
Jul 28 01:14:43.013855 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:14:43.013813    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b571a4_4729_457f_af4a_359f7522a8ec.slice/crio-a39c129b89c62fac4255fe83c1b2f712a13942106153928a31fd27210c4349cd WatchSource:0}: Error finding container a39c129b89c62fac4255fe83c1b2f712a13942106153928a31fd27210c4349cd: Status 404 returned error can't find the container with id a39c129b89c62fac4255fe83c1b2f712a13942106153928a31fd27210c4349cd
Jul 28 01:14:43.044300 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:43.044269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt" event={"ID":"58b571a4-4729-457f-af4a-359f7522a8ec","Type":"ContainerStarted","Data":"a39c129b89c62fac4255fe83c1b2f712a13942106153928a31fd27210c4349cd"}
Jul 28 01:14:44.050036 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:44.049994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt" event={"ID":"58b571a4-4729-457f-af4a-359f7522a8ec","Type":"ContainerStarted","Data":"b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e"}
Jul 28 01:14:44.050437 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:44.050193    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:44.070738 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:44.070680    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt" podStartSLOduration=1.731312872 podStartE2EDuration="2.070664314s" totalImagesPullingTime="339.351442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:14:44.069041354 +0000 UTC m=+1715.624544532" watchObservedRunningTime="2026-07-28 01:14:44.070664314 +0000 UTC m=+1715.626167473"
Jul 28 01:14:45.898088 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:45.898055    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:14:50.061020 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:50.060983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:14:54.112253 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:54.112158    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f99168c-2c8b-4a38-a135-a2610532faac" containerID="c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42" exitCode=0
Jul 28 01:14:54.112253 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:54.112219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" event={"ID":"6f99168c-2c8b-4a38-a135-a2610532faac","Type":"ContainerDied","Data":"c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"}
Jul 28 01:14:54.112683 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:54.112583    2564 scope.go:117] "RemoveContainer" containerID="c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"
Jul 28 01:14:54.113568 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:54.113535    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:14:54.113633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:54.113604    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:14:54.113671 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:54.113636    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:14:55.121386 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:55.121337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" event={"ID":"6f99168c-2c8b-4a38-a135-a2610532faac","Type":"ContainerStarted","Data":"66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798"}
Jul 28 01:14:55.279786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:55.279752    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:55.281045 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:55.281014    2564 patch_prober.go:28] interesting pod/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 28 01:14:55.281258 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:55.281232    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 28 01:14:56.280731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:56.280287    2564 patch_prober.go:28] interesting pod/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 28 01:14:56.280731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:56.280357    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 28 01:14:57.037914 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:57.037877    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:57.279872 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:57.279831    2564 patch_prober.go:28] interesting pod/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 28 01:14:57.280185 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:57.280158    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 28 01:14:58.280089 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:58.280044    2564 patch_prober.go:28] interesting pod/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 28 01:14:58.280652 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:58.280112    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 28 01:14:59.283909 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:59.283875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:14:59.285030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:14:59.285009    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:15:00.134899 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.134864    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"]
Jul 28 01:15:00.142564 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.142536    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"]
Jul 28 01:15:00.142725 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.142659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"
Jul 28 01:15:00.144964 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.144768    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:15:00.146639 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.146613    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"]
Jul 28 01:15:00.146756 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.146725    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"
Jul 28 01:15:00.147669 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.147649    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"]
Jul 28 01:15:00.157328 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.157307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"
Jul 28 01:15:00.164257 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.164235    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"
Jul 28 01:15:00.332250 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.332224    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"]
Jul 28 01:15:00.334491 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:15:00.334467    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399fe7df_5815_42cc_9832_10d74047a955.slice/crio-35d127c98fb483b801b54e7c20e1dc0da6d5cd49e81345a417178c253e4e70f9 WatchSource:0}: Error finding container 35d127c98fb483b801b54e7c20e1dc0da6d5cd49e81345a417178c253e4e70f9: Status 404 returned error can't find the container with id 35d127c98fb483b801b54e7c20e1dc0da6d5cd49e81345a417178c253e4e70f9
Jul 28 01:15:00.512246 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:00.512220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"]
Jul 28 01:15:00.513683 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:15:00.513652    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d153bdc_5919_46ea_97db_1c8e78bf70f6.slice/crio-2bdd0c9269b9eae2f18ad119cc02055bd62d83b5abcb1605d6a67bd895a7fa4e WatchSource:0}: Error finding container 2bdd0c9269b9eae2f18ad119cc02055bd62d83b5abcb1605d6a67bd895a7fa4e: Status 404 returned error can't find the container with id 2bdd0c9269b9eae2f18ad119cc02055bd62d83b5abcb1605d6a67bd895a7fa4e
Jul 28 01:15:01.151977 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:01.151878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q" event={"ID":"399fe7df-5815-42cc-9832-10d74047a955","Type":"ContainerStarted","Data":"57c04ffc729429a61d8b04d81ed5d8cfd170b202458dc99e0fed0af8eeb9cebb"}
Jul 28 01:15:01.151977 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:01.151930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q" event={"ID":"399fe7df-5815-42cc-9832-10d74047a955","Type":"ContainerStarted","Data":"35d127c98fb483b801b54e7c20e1dc0da6d5cd49e81345a417178c253e4e70f9"}
Jul 28 01:15:01.153510 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:01.153486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5" event={"ID":"5d153bdc-5919-46ea-97db-1c8e78bf70f6","Type":"ContainerStarted","Data":"f515f04fc6e15cb5c058adc6f46ed862aa2a9ca6f9b58cc41b479a9721846fcb"}
Jul 28 01:15:01.153613 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:01.153533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5" event={"ID":"5d153bdc-5919-46ea-97db-1c8e78bf70f6","Type":"ContainerStarted","Data":"2bdd0c9269b9eae2f18ad119cc02055bd62d83b5abcb1605d6a67bd895a7fa4e"}
Jul 28 01:15:01.166995 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:01.166892    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q" podStartSLOduration=1.166873079 podStartE2EDuration="1.166873079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:01.166772123 +0000 UTC m=+1732.722275304" watchObservedRunningTime="2026-07-28 01:15:01.166873079 +0000 UTC m=+1732.722376240"
Jul 28 01:15:01.183851 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:01.183789    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5" podStartSLOduration=1.183767289 podStartE2EDuration="1.183767289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:01.182267509 +0000 UTC m=+1732.737770670" watchObservedRunningTime="2026-07-28 01:15:01.183767289 +0000 UTC m=+1732.739270449"
Jul 28 01:15:02.165185 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:02.165146    2564 generic.go:356] "Generic (PLEG): container finished" podID="399fe7df-5815-42cc-9832-10d74047a955" containerID="57c04ffc729429a61d8b04d81ed5d8cfd170b202458dc99e0fed0af8eeb9cebb" exitCode=0
Jul 28 01:15:02.165602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:02.165227    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q" event={"ID":"399fe7df-5815-42cc-9832-10d74047a955","Type":"ContainerDied","Data":"57c04ffc729429a61d8b04d81ed5d8cfd170b202458dc99e0fed0af8eeb9cebb"}
Jul 28 01:15:02.166567 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:02.166546    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d153bdc-5919-46ea-97db-1c8e78bf70f6" containerID="f515f04fc6e15cb5c058adc6f46ed862aa2a9ca6f9b58cc41b479a9721846fcb" exitCode=0
Jul 28 01:15:02.166673 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:02.166575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5" event={"ID":"5d153bdc-5919-46ea-97db-1c8e78bf70f6","Type":"ContainerDied","Data":"f515f04fc6e15cb5c058adc6f46ed862aa2a9ca6f9b58cc41b479a9721846fcb"}
Jul 28 01:15:03.323594 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:03.323568    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"
Jul 28 01:15:03.349969 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:03.349927    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"
Jul 28 01:15:04.176892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:04.176863    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"
Jul 28 01:15:04.176892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:04.176890    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q" event={"ID":"399fe7df-5815-42cc-9832-10d74047a955","Type":"ContainerDied","Data":"35d127c98fb483b801b54e7c20e1dc0da6d5cd49e81345a417178c253e4e70f9"}
Jul 28 01:15:04.177179 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:04.176962    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d127c98fb483b801b54e7c20e1dc0da6d5cd49e81345a417178c253e4e70f9"
Jul 28 01:15:04.178146 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:04.178124    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5" event={"ID":"5d153bdc-5919-46ea-97db-1c8e78bf70f6","Type":"ContainerDied","Data":"2bdd0c9269b9eae2f18ad119cc02055bd62d83b5abcb1605d6a67bd895a7fa4e"}
Jul 28 01:15:04.178246 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:04.178149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753355-nfdz5"
Jul 28 01:15:04.178246 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:04.178158    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bdd0c9269b9eae2f18ad119cc02055bd62d83b5abcb1605d6a67bd895a7fa4e"
Jul 28 01:15:07.463657 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.463618    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"]
Jul 28 01:15:07.467468 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.464580    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399fe7df-5815-42cc-9832-10d74047a955" containerName="cleanup"
Jul 28 01:15:07.467468 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.464625    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="399fe7df-5815-42cc-9832-10d74047a955" containerName="cleanup"
Jul 28 01:15:07.467468 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.464664    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d153bdc-5919-46ea-97db-1c8e78bf70f6" containerName="cleanup"
Jul 28 01:15:07.467468 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.464673    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d153bdc-5919-46ea-97db-1c8e78bf70f6" containerName="cleanup"
Jul 28 01:15:07.467468 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.464882    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d153bdc-5919-46ea-97db-1c8e78bf70f6" containerName="cleanup"
Jul 28 01:15:07.467468 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.464900    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="399fe7df-5815-42cc-9832-10d74047a955" containerName="cleanup"
Jul 28 01:15:07.471408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.471355    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:07.473680 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.473655    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:15:07.473838 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.473731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"]
Jul 28 01:15:07.536154 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.536120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5518d447-0983-4698-a020-5bd33df46148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf\" (UID: \"5518d447-0983-4698-a020-5bd33df46148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:07.637486 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.637442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5518d447-0983-4698-a020-5bd33df46148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf\" (UID: \"5518d447-0983-4698-a020-5bd33df46148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:07.638114 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.638092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5518d447-0983-4698-a020-5bd33df46148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf\" (UID: \"5518d447-0983-4698-a020-5bd33df46148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:07.783823 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.783703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:07.913333 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:07.913307    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"]
Jul 28 01:15:07.916060 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:15:07.916029    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5518d447_0983_4698_a020_5bd33df46148.slice/crio-33ac9ca61813f659462494599be72cf59582c32026161aea9d8bc3800ccc49da WatchSource:0}: Error finding container 33ac9ca61813f659462494599be72cf59582c32026161aea9d8bc3800ccc49da: Status 404 returned error can't find the container with id 33ac9ca61813f659462494599be72cf59582c32026161aea9d8bc3800ccc49da
Jul 28 01:15:08.199439 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:08.199405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf" event={"ID":"5518d447-0983-4698-a020-5bd33df46148","Type":"ContainerStarted","Data":"0ec13282d9dc2e9dab54a852ebe7d7f99f4d8741bbda847d4221947a6f87fe5d"}
Jul 28 01:15:08.199594 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:08.199461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf" event={"ID":"5518d447-0983-4698-a020-5bd33df46148","Type":"ContainerStarted","Data":"33ac9ca61813f659462494599be72cf59582c32026161aea9d8bc3800ccc49da"}
Jul 28 01:15:08.213952 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:08.213887    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf" podStartSLOduration=1.213873349 podStartE2EDuration="1.213873349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:08.212789194 +0000 UTC m=+1739.768292354" watchObservedRunningTime="2026-07-28 01:15:08.213873349 +0000 UTC m=+1739.769376509"
Jul 28 01:15:09.206174 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:09.206140    2564 generic.go:356] "Generic (PLEG): container finished" podID="5518d447-0983-4698-a020-5bd33df46148" containerID="0ec13282d9dc2e9dab54a852ebe7d7f99f4d8741bbda847d4221947a6f87fe5d" exitCode=0
Jul 28 01:15:09.206545 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:09.206190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf" event={"ID":"5518d447-0983-4698-a020-5bd33df46148","Type":"ContainerDied","Data":"0ec13282d9dc2e9dab54a852ebe7d7f99f4d8741bbda847d4221947a6f87fe5d"}
Jul 28 01:15:10.346151 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:10.346124    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:10.357584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:10.357558    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5518d447-0983-4698-a020-5bd33df46148-maas-api-service-ca\") pod \"5518d447-0983-4698-a020-5bd33df46148\" (UID: \"5518d447-0983-4698-a020-5bd33df46148\") "
Jul 28 01:15:10.357920 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:10.357894    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5518d447-0983-4698-a020-5bd33df46148-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5518d447-0983-4698-a020-5bd33df46148" (UID: "5518d447-0983-4698-a020-5bd33df46148"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:15:10.458159 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:10.458125    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5518d447-0983-4698-a020-5bd33df46148-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:11.216882 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:11.216793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf" event={"ID":"5518d447-0983-4698-a020-5bd33df46148","Type":"ContainerDied","Data":"33ac9ca61813f659462494599be72cf59582c32026161aea9d8bc3800ccc49da"}
Jul 28 01:15:11.216882 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:11.216830    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"
Jul 28 01:15:11.216882 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:11.216846    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ac9ca61813f659462494599be72cf59582c32026161aea9d8bc3800ccc49da"
Jul 28 01:15:17.977628 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:17.977595    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"]
Jul 28 01:15:17.978088 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:17.977822    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt" podUID="58b571a4-4729-457f-af4a-359f7522a8ec" containerName="maas-api" containerID="cri-o://b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e" gracePeriod=30
Jul 28 01:15:17.997365 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:17.997342    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"]
Jul 28 01:15:18.003605 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.003580    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d990b63a-29753355-8tn9q"]
Jul 28 01:15:18.236458 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.236389    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:15:18.251132 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.251097    2564 generic.go:356] "Generic (PLEG): container finished" podID="58b571a4-4729-457f-af4a-359f7522a8ec" containerID="b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e" exitCode=0
Jul 28 01:15:18.251261 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.251138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt" event={"ID":"58b571a4-4729-457f-af4a-359f7522a8ec","Type":"ContainerDied","Data":"b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e"}
Jul 28 01:15:18.251261 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.251143    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"
Jul 28 01:15:18.251261 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.251231    2564 scope.go:117] "RemoveContainer" containerID="b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e"
Jul 28 01:15:18.251374 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.251219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt" event={"ID":"58b571a4-4729-457f-af4a-359f7522a8ec","Type":"ContainerDied","Data":"a39c129b89c62fac4255fe83c1b2f712a13942106153928a31fd27210c4349cd"}
Jul 28 01:15:18.263266 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.263242    2564 scope.go:117] "RemoveContainer" containerID="b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e"
Jul 28 01:15:18.263599 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:15:18.263578    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e\": container with ID starting with b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e not found: ID does not exist" containerID="b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e"
Jul 28 01:15:18.263684 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.263607    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e"} err="failed to get container status \"b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e\": rpc error: code = NotFound desc = could not find container \"b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e\": container with ID starting with b2402ed2e1be4113e58ec96ac502597de17446ab5ac8dbd2551edc34b32dd52e not found: ID does not exist"
Jul 28 01:15:18.329478 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.329443    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b571a4-4729-457f-af4a-359f7522a8ec-openshift-service-ca\") pod \"58b571a4-4729-457f-af4a-359f7522a8ec\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") "
Jul 28 01:15:18.329668 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.329530    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twcn8\" (UniqueName: \"kubernetes.io/projected/58b571a4-4729-457f-af4a-359f7522a8ec-kube-api-access-twcn8\") pod \"58b571a4-4729-457f-af4a-359f7522a8ec\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") "
Jul 28 01:15:18.329668 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.329566    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b571a4-4729-457f-af4a-359f7522a8ec-maas-api-tls\") pod \"58b571a4-4729-457f-af4a-359f7522a8ec\" (UID: \"58b571a4-4729-457f-af4a-359f7522a8ec\") "
Jul 28 01:15:18.329930 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.329849    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b571a4-4729-457f-af4a-359f7522a8ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58b571a4-4729-457f-af4a-359f7522a8ec" (UID: "58b571a4-4729-457f-af4a-359f7522a8ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:15:18.332471 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.332446    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b571a4-4729-457f-af4a-359f7522a8ec-kube-api-access-twcn8" (OuterVolumeSpecName: "kube-api-access-twcn8") pod "58b571a4-4729-457f-af4a-359f7522a8ec" (UID: "58b571a4-4729-457f-af4a-359f7522a8ec"). InnerVolumeSpecName "kube-api-access-twcn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:15:18.332471 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.332444    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b571a4-4729-457f-af4a-359f7522a8ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58b571a4-4729-457f-af4a-359f7522a8ec" (UID: "58b571a4-4729-457f-af4a-359f7522a8ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:15:18.431104 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.431066    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcn8\" (UniqueName: \"kubernetes.io/projected/58b571a4-4729-457f-af4a-359f7522a8ec-kube-api-access-twcn8\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:18.431104 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.431100    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b571a4-4729-457f-af4a-359f7522a8ec-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:18.431104 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.431110    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b571a4-4729-457f-af4a-359f7522a8ec-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:18.573604 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.573571    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"]
Jul 28 01:15:18.577629 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.577600    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d990b63a-68f966cb69-zlprt"]
Jul 28 01:15:18.944889 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.944850    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="399fe7df-5815-42cc-9832-10d74047a955" path="/var/lib/kubelet/pods/399fe7df-5815-42cc-9832-10d74047a955/volumes"
Jul 28 01:15:18.945278 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:18.945259    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b571a4-4729-457f-af4a-359f7522a8ec" path="/var/lib/kubelet/pods/58b571a4-4729-457f-af4a-359f7522a8ec/volumes"
Jul 28 01:15:21.094038 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:21.093996    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:15:22.545779 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:22.545723    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"]
Jul 28 01:15:22.547966 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:22.547922    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d990b63a-g9srf"]
Jul 28 01:15:22.685099 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:22.685063    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"]
Jul 28 01:15:22.685390 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:22.685361    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy" containerID="cri-o://66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798" gracePeriod=30
Jul 28 01:15:22.944750 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:22.944720    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5518d447-0983-4698-a020-5bd33df46148" path="/var/lib/kubelet/pods/5518d447-0983-4698-a020-5bd33df46148/volumes"
Jul 28 01:15:25.875383 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.875344    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"]
Jul 28 01:15:25.875892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.875873    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b571a4-4729-457f-af4a-359f7522a8ec" containerName="maas-api"
Jul 28 01:15:25.876003 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.875894    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b571a4-4729-457f-af4a-359f7522a8ec" containerName="maas-api"
Jul 28 01:15:25.876003 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.875916    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5518d447-0983-4698-a020-5bd33df46148" containerName="revoke-keys"
Jul 28 01:15:25.876003 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.875925    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5518d447-0983-4698-a020-5bd33df46148" containerName="revoke-keys"
Jul 28 01:15:25.876141 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.876034    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5518d447-0983-4698-a020-5bd33df46148" containerName="revoke-keys"
Jul 28 01:15:25.876141 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.876050    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b571a4-4729-457f-af4a-359f7522a8ec" containerName="maas-api"
Jul 28 01:15:25.880347 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.880326    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:25.889165 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.889141    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-60243cce-serving-cert\""
Jul 28 01:15:25.889286 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.889210    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:15:25.890107 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.890088    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:15:25.890194 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.890089    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:15:25.896412 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.896387    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"]
Jul 28 01:15:25.997758 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.997715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd93667b-2984-4020-9de8-813a982ea938-openshift-service-ca\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:25.997992 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.997784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfss\" (UniqueName: \"kubernetes.io/projected/dd93667b-2984-4020-9de8-813a982ea938-kube-api-access-jsfss\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:25.997992 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:25.997836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd93667b-2984-4020-9de8-813a982ea938-maas-api-tls\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.099142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.099103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd93667b-2984-4020-9de8-813a982ea938-openshift-service-ca\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.099142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.099143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfss\" (UniqueName: \"kubernetes.io/projected/dd93667b-2984-4020-9de8-813a982ea938-kube-api-access-jsfss\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.099356 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.099185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd93667b-2984-4020-9de8-813a982ea938-maas-api-tls\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.099740 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.099710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd93667b-2984-4020-9de8-813a982ea938-openshift-service-ca\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.101585 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.101560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd93667b-2984-4020-9de8-813a982ea938-maas-api-tls\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.109428 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.109399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfss\" (UniqueName: \"kubernetes.io/projected/dd93667b-2984-4020-9de8-813a982ea938-kube-api-access-jsfss\") pod \"maas-api-e2e-ait-60243cce-7686948d58-x8vmf\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.191365 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.191326    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:26.331792 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:26.331765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"]
Jul 28 01:15:26.333082 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:15:26.333049    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd93667b_2984_4020_9de8_813a982ea938.slice/crio-21eeedc6e2967617f08b221a0a653b94e62fe30c915bb997212e90ecd69cea7f WatchSource:0}: Error finding container 21eeedc6e2967617f08b221a0a653b94e62fe30c915bb997212e90ecd69cea7f: Status 404 returned error can't find the container with id 21eeedc6e2967617f08b221a0a653b94e62fe30c915bb997212e90ecd69cea7f
Jul 28 01:15:27.298161 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:27.298131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf" event={"ID":"dd93667b-2984-4020-9de8-813a982ea938","Type":"ContainerStarted","Data":"97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565"}
Jul 28 01:15:27.298633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:27.298189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf" event={"ID":"dd93667b-2984-4020-9de8-813a982ea938","Type":"ContainerStarted","Data":"21eeedc6e2967617f08b221a0a653b94e62fe30c915bb997212e90ecd69cea7f"}
Jul 28 01:15:27.298633 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:27.298220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:27.318349 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:27.318226    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf" podStartSLOduration=1.588869956 podStartE2EDuration="2.318204481s" totalImagesPullingTime="729.334525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:27.316423274 +0000 UTC m=+1758.871926458" watchObservedRunningTime="2026-07-28 01:15:27.318204481 +0000 UTC m=+1758.873707642"
Jul 28 01:15:27.936910 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:27.936886    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:15:28.017100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-credential-socket\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017056    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-istio-token\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017079    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-socket\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-envoy\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017203    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zsck\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-kube-api-access-7zsck\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017233    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99168c-2c8b-4a38-a135-a2610532faac-istiod-ca-cert\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017290    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99168c-2c8b-4a38-a135-a2610532faac-istio-podinfo\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017331    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-data\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:15:28.017584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017376    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-certs\") pod \"6f99168c-2c8b-4a38-a135-a2610532faac\" (UID: \"6f99168c-2c8b-4a38-a135-a2610532faac\") "
Jul 28 01:15:28.017584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017409    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:15:28.017708 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:15:28.017794 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017743    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.017794 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017782    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.017875 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017800    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.017875 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.017836    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f99168c-2c8b-4a38-a135-a2610532faac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:15:28.018348 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.018323    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:15:28.019547 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.019521    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:15:28.019675 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.019551    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-kube-api-access-7zsck" (OuterVolumeSpecName: "kube-api-access-7zsck") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "kube-api-access-7zsck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:15:28.019675 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.019648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f99168c-2c8b-4a38-a135-a2610532faac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:15:28.019675 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.019668    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f99168c-2c8b-4a38-a135-a2610532faac" (UID: "6f99168c-2c8b-4a38-a135-a2610532faac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:15:28.118322 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.118292    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.118322 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.118318    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zsck\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-kube-api-access-7zsck\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.118322 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.118326    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99168c-2c8b-4a38-a135-a2610532faac-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.118552 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.118335    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99168c-2c8b-4a38-a135-a2610532faac-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.118552 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.118343    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99168c-2c8b-4a38-a135-a2610532faac-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.118552 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.118352    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99168c-2c8b-4a38-a135-a2610532faac-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:28.304996 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.304892    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f99168c-2c8b-4a38-a135-a2610532faac" containerID="66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798" exitCode=0
Jul 28 01:15:28.304996 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.304966    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"
Jul 28 01:15:28.304996 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.304972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" event={"ID":"6f99168c-2c8b-4a38-a135-a2610532faac","Type":"ContainerDied","Data":"66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798"}
Jul 28 01:15:28.305526 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.305029    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r" event={"ID":"6f99168c-2c8b-4a38-a135-a2610532faac","Type":"ContainerDied","Data":"200410ae9dc4fc0c446ae80f9a50fed1e875a6712ab14e858ae05d606fb9df49"}
Jul 28 01:15:28.305526 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.305054    2564 scope.go:117] "RemoveContainer" containerID="66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798"
Jul 28 01:15:28.314343 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.314317    2564 scope.go:117] "RemoveContainer" containerID="c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"
Jul 28 01:15:28.322668 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.322646    2564 scope.go:117] "RemoveContainer" containerID="66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798"
Jul 28 01:15:28.322974 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:15:28.322952    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798\": container with ID starting with 66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798 not found: ID does not exist" containerID="66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798"
Jul 28 01:15:28.323031 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.322997    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798"} err="failed to get container status \"66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798\": rpc error: code = NotFound desc = could not find container \"66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798\": container with ID starting with 66dd87f22aae163184960048cf230cbae75ab5ee0915b08bea617d37fd7e4798 not found: ID does not exist"
Jul 28 01:15:28.323031 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.323019    2564 scope.go:117] "RemoveContainer" containerID="c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"
Jul 28 01:15:28.323273 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:15:28.323255    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42\": container with ID starting with c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42 not found: ID does not exist" containerID="c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"
Jul 28 01:15:28.323311 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.323279    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42"} err="failed to get container status \"c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42\": rpc error: code = NotFound desc = could not find container \"c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42\": container with ID starting with c2e41ccf5121afd5ab17afa88200a37aa20ec63ec9275ed3ab6ef27d75438b42 not found: ID does not exist"
Jul 28 01:15:28.326051 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.326025    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"]
Jul 28 01:15:28.330009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.329982    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d990b63a-gw-openshift-default-75548b4676-srr7r"]
Jul 28 01:15:28.949355 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:28.949323    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" path="/var/lib/kubelet/pods/6f99168c-2c8b-4a38-a135-a2610532faac/volumes"
Jul 28 01:15:32.115029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:32.114987    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:15:33.310561 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:33.310526    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:15:51.455648 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.455608    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"]
Jul 28 01:15:51.456064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.455982    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy"
Jul 28 01:15:51.456064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.455995    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy"
Jul 28 01:15:51.456064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.456003    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy"
Jul 28 01:15:51.456064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.456008    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy"
Jul 28 01:15:51.456064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.456066    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy"
Jul 28 01:15:51.456225 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.456076    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f99168c-2c8b-4a38-a135-a2610532faac" containerName="istio-proxy"
Jul 28 01:15:51.459007 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.458991    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:51.461269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.461248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:15:51.464858 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.464836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"]
Jul 28 01:15:51.524544 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.524507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cd0dd0-5fce-49a0-b7e5-86249fe8003e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-60243cce-t2lxn\" (UID: \"84cd0dd0-5fce-49a0-b7e5-86249fe8003e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:51.625482 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.625446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cd0dd0-5fce-49a0-b7e5-86249fe8003e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-60243cce-t2lxn\" (UID: \"84cd0dd0-5fce-49a0-b7e5-86249fe8003e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:51.626081 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.626061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cd0dd0-5fce-49a0-b7e5-86249fe8003e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-60243cce-t2lxn\" (UID: \"84cd0dd0-5fce-49a0-b7e5-86249fe8003e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:51.770835 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:51.770753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:52.110961 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:52.110910    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"]
Jul 28 01:15:52.113178 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:15:52.113150    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cd0dd0_5fce_49a0_b7e5_86249fe8003e.slice/crio-54c1f2c4e72e18d700a7786317b6f9dc2484344c47795b0076dbdef3cff374f4 WatchSource:0}: Error finding container 54c1f2c4e72e18d700a7786317b6f9dc2484344c47795b0076dbdef3cff374f4: Status 404 returned error can't find the container with id 54c1f2c4e72e18d700a7786317b6f9dc2484344c47795b0076dbdef3cff374f4
Jul 28 01:15:52.427329 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:52.427289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn" event={"ID":"84cd0dd0-5fce-49a0-b7e5-86249fe8003e","Type":"ContainerStarted","Data":"6e90ca8f4f69c21e5fdc21b7b93c799fda9f746a752837f818745dbe967534ae"}
Jul 28 01:15:52.427493 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:52.427350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn" event={"ID":"84cd0dd0-5fce-49a0-b7e5-86249fe8003e","Type":"ContainerStarted","Data":"54c1f2c4e72e18d700a7786317b6f9dc2484344c47795b0076dbdef3cff374f4"}
Jul 28 01:15:52.441707 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:52.441654    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn" podStartSLOduration=1.441638486 podStartE2EDuration="1.441638486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:52.440669135 +0000 UTC m=+1783.996172302" watchObservedRunningTime="2026-07-28 01:15:52.441638486 +0000 UTC m=+1783.997141646"
Jul 28 01:15:53.433968 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:53.433914    2564 generic.go:356] "Generic (PLEG): container finished" podID="84cd0dd0-5fce-49a0-b7e5-86249fe8003e" containerID="6e90ca8f4f69c21e5fdc21b7b93c799fda9f746a752837f818745dbe967534ae" exitCode=0
Jul 28 01:15:53.433968 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:53.433962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn" event={"ID":"84cd0dd0-5fce-49a0-b7e5-86249fe8003e","Type":"ContainerDied","Data":"6e90ca8f4f69c21e5fdc21b7b93c799fda9f746a752837f818745dbe967534ae"}
Jul 28 01:15:54.581356 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:54.581328    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:54.646239 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:54.646201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cd0dd0-5fce-49a0-b7e5-86249fe8003e-maas-api-service-ca\") pod \"84cd0dd0-5fce-49a0-b7e5-86249fe8003e\" (UID: \"84cd0dd0-5fce-49a0-b7e5-86249fe8003e\") "
Jul 28 01:15:54.646568 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:54.646543    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cd0dd0-5fce-49a0-b7e5-86249fe8003e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84cd0dd0-5fce-49a0-b7e5-86249fe8003e" (UID: "84cd0dd0-5fce-49a0-b7e5-86249fe8003e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:15:54.747770 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:54.747677    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cd0dd0-5fce-49a0-b7e5-86249fe8003e-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:15:55.445422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:55.445389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn" event={"ID":"84cd0dd0-5fce-49a0-b7e5-86249fe8003e","Type":"ContainerDied","Data":"54c1f2c4e72e18d700a7786317b6f9dc2484344c47795b0076dbdef3cff374f4"}
Jul 28 01:15:55.445422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:55.445416    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"
Jul 28 01:15:55.445634 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:15:55.445438    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54c1f2c4e72e18d700a7786317b6f9dc2484344c47795b0076dbdef3cff374f4"
Jul 28 01:16:01.848070 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:01.847970    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"]
Jul 28 01:16:01.848566 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:01.848222    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf" podUID="dd93667b-2984-4020-9de8-813a982ea938" containerName="maas-api" containerID="cri-o://97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565" gracePeriod=30
Jul 28 01:16:02.104792 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.104730    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:16:02.217890 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.217858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd93667b-2984-4020-9de8-813a982ea938-openshift-service-ca\") pod \"dd93667b-2984-4020-9de8-813a982ea938\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") "
Jul 28 01:16:02.218152 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.217911    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsfss\" (UniqueName: \"kubernetes.io/projected/dd93667b-2984-4020-9de8-813a982ea938-kube-api-access-jsfss\") pod \"dd93667b-2984-4020-9de8-813a982ea938\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") "
Jul 28 01:16:02.218152 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.217991    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd93667b-2984-4020-9de8-813a982ea938-maas-api-tls\") pod \"dd93667b-2984-4020-9de8-813a982ea938\" (UID: \"dd93667b-2984-4020-9de8-813a982ea938\") "
Jul 28 01:16:02.218416 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.218389    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd93667b-2984-4020-9de8-813a982ea938-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd93667b-2984-4020-9de8-813a982ea938" (UID: "dd93667b-2984-4020-9de8-813a982ea938"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:02.220263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.220232    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd93667b-2984-4020-9de8-813a982ea938-kube-api-access-jsfss" (OuterVolumeSpecName: "kube-api-access-jsfss") pod "dd93667b-2984-4020-9de8-813a982ea938" (UID: "dd93667b-2984-4020-9de8-813a982ea938"). InnerVolumeSpecName "kube-api-access-jsfss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:02.220263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.220240    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd93667b-2984-4020-9de8-813a982ea938-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd93667b-2984-4020-9de8-813a982ea938" (UID: "dd93667b-2984-4020-9de8-813a982ea938"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:16:02.318864 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.318824    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd93667b-2984-4020-9de8-813a982ea938-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.318864 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.318860    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsfss\" (UniqueName: \"kubernetes.io/projected/dd93667b-2984-4020-9de8-813a982ea938-kube-api-access-jsfss\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.318864 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.318870    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd93667b-2984-4020-9de8-813a982ea938-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.481055 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.481024    2564 generic.go:356] "Generic (PLEG): container finished" podID="dd93667b-2984-4020-9de8-813a982ea938" containerID="97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565" exitCode=0
Jul 28 01:16:02.481233 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.481074    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"
Jul 28 01:16:02.481233 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.481114    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf" event={"ID":"dd93667b-2984-4020-9de8-813a982ea938","Type":"ContainerDied","Data":"97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565"}
Jul 28 01:16:02.481233 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.481201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf" event={"ID":"dd93667b-2984-4020-9de8-813a982ea938","Type":"ContainerDied","Data":"21eeedc6e2967617f08b221a0a653b94e62fe30c915bb997212e90ecd69cea7f"}
Jul 28 01:16:02.481336 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.481234    2564 scope.go:117] "RemoveContainer" containerID="97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565"
Jul 28 01:16:02.490472 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.490448    2564 scope.go:117] "RemoveContainer" containerID="97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565"
Jul 28 01:16:02.490763 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:16:02.490744    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565\": container with ID starting with 97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565 not found: ID does not exist" containerID="97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565"
Jul 28 01:16:02.490825 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.490772    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565"} err="failed to get container status \"97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565\": rpc error: code = NotFound desc = could not find container \"97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565\": container with ID starting with 97c663a6f42a1e382b95a8b6e772f47c65eb3e25817146352c0230a28e12c565 not found: ID does not exist"
Jul 28 01:16:02.504210 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.504178    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"]
Jul 28 01:16:02.509264 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.509235    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60243cce-7686948d58-x8vmf"]
Jul 28 01:16:02.946826 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:02.946792    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd93667b-2984-4020-9de8-813a982ea938" path="/var/lib/kubelet/pods/dd93667b-2984-4020-9de8-813a982ea938/volumes"
Jul 28 01:16:04.702494 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:04.702461    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:16:06.522545 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:06.522500    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"]
Jul 28 01:16:06.525501 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:06.525469    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60243cce-t2lxn"]
Jul 28 01:16:06.947079 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:06.945387    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84cd0dd0-5fce-49a0-b7e5-86249fe8003e" path="/var/lib/kubelet/pods/84cd0dd0-5fce-49a0-b7e5-86249fe8003e/volumes"
Jul 28 01:16:08.492353 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492311    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"]
Jul 28 01:16:08.492731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492673    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84cd0dd0-5fce-49a0-b7e5-86249fe8003e" containerName="revoke-keys"
Jul 28 01:16:08.492731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492684    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84cd0dd0-5fce-49a0-b7e5-86249fe8003e" containerName="revoke-keys"
Jul 28 01:16:08.492731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492705    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd93667b-2984-4020-9de8-813a982ea938" containerName="maas-api"
Jul 28 01:16:08.492731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492713    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd93667b-2984-4020-9de8-813a982ea938" containerName="maas-api"
Jul 28 01:16:08.492872 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492775    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84cd0dd0-5fce-49a0-b7e5-86249fe8003e" containerName="revoke-keys"
Jul 28 01:16:08.492872 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.492785    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd93667b-2984-4020-9de8-813a982ea938" containerName="maas-api"
Jul 28 01:16:08.495794 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.495773    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.498785 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.498762    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f6442e46-openshift-default-dockercfg-s9xbr\""
Jul 28 01:16:08.514352 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.514322    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"]
Jul 28 01:16:08.570722 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-envoy\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.570892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-istio-token\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.570892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-data\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.570892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgb5\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-kube-api-access-kfgb5\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.570892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570882    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-socket\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.571115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d022b15-a124-4332-94d7-72a57de38eae-istio-podinfo\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.571115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.570953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-certs\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.571115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.571054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d022b15-a124-4332-94d7-72a57de38eae-istiod-ca-cert\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.571115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.571106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-credential-socket\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.671993 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.671951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-envoy\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672175 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672001    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-istio-token\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672175 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-data\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672175 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgb5\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-kube-api-access-kfgb5\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672175 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-socket\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672175 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d022b15-a124-4332-94d7-72a57de38eae-istio-podinfo\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672418 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-certs\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672418 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672253    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d022b15-a124-4332-94d7-72a57de38eae-istiod-ca-cert\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672418 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-credential-socket\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672569 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-data\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672569 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-socket\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672774 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-certs\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.672920 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.672895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-credential-socket\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.673090 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.673073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d022b15-a124-4332-94d7-72a57de38eae-istiod-ca-cert\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.674619 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.674595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d022b15-a124-4332-94d7-72a57de38eae-istio-podinfo\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.674706 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.674623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-envoy\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.680278 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.680251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-istio-token\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.680403 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.680316    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgb5\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-kube-api-access-kfgb5\") pod \"e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") " pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.806384 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.806299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:08.987440 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.987367    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"]
Jul 28 01:16:08.989387 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:16:08.989336    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d022b15_a124_4332_94d7_72a57de38eae.slice/crio-1efcf042622a46790575d335046b6cb1a48b4f9259a3469cac2994d7de0125a5 WatchSource:0}: Error finding container 1efcf042622a46790575d335046b6cb1a48b4f9259a3469cac2994d7de0125a5: Status 404 returned error can't find the container with id 1efcf042622a46790575d335046b6cb1a48b4f9259a3469cac2994d7de0125a5
Jul 28 01:16:08.991814 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.991772    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:16:08.991914 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.991862    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:16:08.991914 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.991904    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:16:08.998209 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.998186    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:16:08.999235 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:08.999218    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:16:09.517351 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:09.517311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" event={"ID":"4d022b15-a124-4332-94d7-72a57de38eae","Type":"ContainerStarted","Data":"ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"}
Jul 28 01:16:09.517761 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:09.517367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" event={"ID":"4d022b15-a124-4332-94d7-72a57de38eae","Type":"ContainerStarted","Data":"1efcf042622a46790575d335046b6cb1a48b4f9259a3469cac2994d7de0125a5"}
Jul 28 01:16:09.537255 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:09.537204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podStartSLOduration=1.537185772 podStartE2EDuration="1.537185772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:09.535455904 +0000 UTC m=+1801.090959066" watchObservedRunningTime="2026-07-28 01:16:09.537185772 +0000 UTC m=+1801.092688933"
Jul 28 01:16:09.807138 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:09.807051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:10.812005 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:10.811973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:11.102021 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.101914    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"]
Jul 28 01:16:11.105375 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.105355    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.107652 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.107630    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f6442e46-serving-cert\""
Jul 28 01:16:11.107793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.107772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:16:11.107971 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.107954    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:16:11.108058 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.107989    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:16:11.116098 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.116054    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"]
Jul 28 01:16:11.198391 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.198346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.198587 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.198422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcnx\" (UniqueName: \"kubernetes.io/projected/5bd41250-7ad4-4d75-91d7-200c62bab5ab-kube-api-access-bhcnx\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.198587 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.198455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd41250-7ad4-4d75-91d7-200c62bab5ab-openshift-service-ca\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.299760 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.299709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.300010 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.299799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcnx\" (UniqueName: \"kubernetes.io/projected/5bd41250-7ad4-4d75-91d7-200c62bab5ab-kube-api-access-bhcnx\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.300010 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.299834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd41250-7ad4-4d75-91d7-200c62bab5ab-openshift-service-ca\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.300010 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:16:11.299905    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-serving-cert: secret "maas-api-e2e-derive-f6442e46-serving-cert" not found
Jul 28 01:16:11.300200 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:16:11.300024    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls podName:5bd41250-7ad4-4d75-91d7-200c62bab5ab nodeName:}" failed. No retries permitted until 2026-07-28 01:16:11.799999107 +0000 UTC m=+1803.355502258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls") pod "maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" (UID: "5bd41250-7ad4-4d75-91d7-200c62bab5ab") : secret "maas-api-e2e-derive-f6442e46-serving-cert" not found
Jul 28 01:16:11.300672 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.300648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd41250-7ad4-4d75-91d7-200c62bab5ab-openshift-service-ca\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.313229 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.313205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcnx\" (UniqueName: \"kubernetes.io/projected/5bd41250-7ad4-4d75-91d7-200c62bab5ab-kube-api-access-bhcnx\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.527704 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.527676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:11.528572 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.528554    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:11.804783 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.804700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:11.807531 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:11.807500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls\") pod \"maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:12.017798 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:12.017745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:12.159310 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:12.159279    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"]
Jul 28 01:16:12.161730 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:16:12.161690    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd41250_7ad4_4d75_91d7_200c62bab5ab.slice/crio-3e7ecbc8b36aea5563e24fc1ae83c5e193677378ceac287a15f1b8a323120366 WatchSource:0}: Error finding container 3e7ecbc8b36aea5563e24fc1ae83c5e193677378ceac287a15f1b8a323120366: Status 404 returned error can't find the container with id 3e7ecbc8b36aea5563e24fc1ae83c5e193677378ceac287a15f1b8a323120366
Jul 28 01:16:12.533816 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:12.533782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" event={"ID":"5bd41250-7ad4-4d75-91d7-200c62bab5ab","Type":"ContainerStarted","Data":"3e7ecbc8b36aea5563e24fc1ae83c5e193677378ceac287a15f1b8a323120366"}
Jul 28 01:16:13.540274 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:13.540236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" event={"ID":"5bd41250-7ad4-4d75-91d7-200c62bab5ab","Type":"ContainerStarted","Data":"e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d"}
Jul 28 01:16:13.540697 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:13.540322    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:13.557601 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:13.557552    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" podStartSLOduration=2.150373493 podStartE2EDuration="2.557537644s" totalImagesPullingTime="407.164151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:13.556050162 +0000 UTC m=+1805.111553342" watchObservedRunningTime="2026-07-28 01:16:13.557537644 +0000 UTC m=+1805.113040804"
Jul 28 01:16:13.902167 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:13.902073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:16:19.550188 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:19.550160    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:22.595307 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:22.595268    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d022b15-a124-4332-94d7-72a57de38eae" containerID="ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658" exitCode=0
Jul 28 01:16:22.595755 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:22.595340    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" event={"ID":"4d022b15-a124-4332-94d7-72a57de38eae","Type":"ContainerDied","Data":"ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"}
Jul 28 01:16:22.595755 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:22.595748    2564 scope.go:117] "RemoveContainer" containerID="ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"
Jul 28 01:16:22.596750 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:22.596724    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:16:22.596812 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:22.596799    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:16:22.596848 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:22.596831    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:16:23.609732 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:23.609692    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" event={"ID":"4d022b15-a124-4332-94d7-72a57de38eae","Type":"ContainerStarted","Data":"79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e"}
Jul 28 01:16:23.806869 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:23.806827    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:23.808163 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:23.808136    2564 patch_prober.go:28] interesting pod/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 28 01:16:23.808268 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:23.808189    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 28 01:16:24.806855 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:24.806815    2564 patch_prober.go:28] interesting pod/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 28 01:16:24.807310 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:24.806890    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 28 01:16:25.807274 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:25.807240    2564 patch_prober.go:28] interesting pod/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 28 01:16:25.807697 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:25.807293    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 28 01:16:26.527889 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:26.527851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:26.807274 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:26.807193    2564 patch_prober.go:28] interesting pod/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 28 01:16:26.807274 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:26.807261    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 28 01:16:27.807035 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:27.806999    2564 patch_prober.go:28] interesting pod/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 28 01:16:27.807219 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:27.807065    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 28 01:16:28.811057 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:28.811022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:28.812006 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:28.811987    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:35.696488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.696447    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"]
Jul 28 01:16:35.700140 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.700117    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:35.702564 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.702542    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:16:35.706409 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.706376    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"]
Jul 28 01:16:35.729382 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.729348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96886c-3ec5-44c6-9560-8631f7da05c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz\" (UID: \"2c96886c-3ec5-44c6-9560-8631f7da05c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:35.830487 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.830449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96886c-3ec5-44c6-9560-8631f7da05c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz\" (UID: \"2c96886c-3ec5-44c6-9560-8631f7da05c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:35.831135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:35.831114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96886c-3ec5-44c6-9560-8631f7da05c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz\" (UID: \"2c96886c-3ec5-44c6-9560-8631f7da05c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:36.013427 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:36.013335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:36.154481 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:36.154442    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"]
Jul 28 01:16:36.155483 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:16:36.155447    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c96886c_3ec5_44c6_9560_8631f7da05c5.slice/crio-0dd28050effa3fc9ac1a911e7b3df908efb5cbbc6daa89d88532cc55b717cbfe WatchSource:0}: Error finding container 0dd28050effa3fc9ac1a911e7b3df908efb5cbbc6daa89d88532cc55b717cbfe: Status 404 returned error can't find the container with id 0dd28050effa3fc9ac1a911e7b3df908efb5cbbc6daa89d88532cc55b717cbfe
Jul 28 01:16:36.675629 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:36.675544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz" event={"ID":"2c96886c-3ec5-44c6-9560-8631f7da05c5","Type":"ContainerStarted","Data":"52c7a9f5f929437fe5f7b65808f0ad5d4c86ff6d31ab4b37f71b68f95c3fa1a0"}
Jul 28 01:16:36.675629 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:36.675596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz" event={"ID":"2c96886c-3ec5-44c6-9560-8631f7da05c5","Type":"ContainerStarted","Data":"0dd28050effa3fc9ac1a911e7b3df908efb5cbbc6daa89d88532cc55b717cbfe"}
Jul 28 01:16:36.690841 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:36.690786    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz" podStartSLOduration=1.690768236 podStartE2EDuration="1.690768236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:36.690103169 +0000 UTC m=+1828.245606329" watchObservedRunningTime="2026-07-28 01:16:36.690768236 +0000 UTC m=+1828.246271396"
Jul 28 01:16:37.681804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:37.681768    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c96886c-3ec5-44c6-9560-8631f7da05c5" containerID="52c7a9f5f929437fe5f7b65808f0ad5d4c86ff6d31ab4b37f71b68f95c3fa1a0" exitCode=0
Jul 28 01:16:37.682260 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:37.681864    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz" event={"ID":"2c96886c-3ec5-44c6-9560-8631f7da05c5","Type":"ContainerDied","Data":"52c7a9f5f929437fe5f7b65808f0ad5d4c86ff6d31ab4b37f71b68f95c3fa1a0"}
Jul 28 01:16:38.832170 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:38.832145    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:38.854617 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:38.854585    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96886c-3ec5-44c6-9560-8631f7da05c5-maas-api-service-ca\") pod \"2c96886c-3ec5-44c6-9560-8631f7da05c5\" (UID: \"2c96886c-3ec5-44c6-9560-8631f7da05c5\") "
Jul 28 01:16:38.854961 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:38.854922    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c96886c-3ec5-44c6-9560-8631f7da05c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c96886c-3ec5-44c6-9560-8631f7da05c5" (UID: "2c96886c-3ec5-44c6-9560-8631f7da05c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:38.955844 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:38.955813    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c96886c-3ec5-44c6-9560-8631f7da05c5-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:39.694093 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:39.694050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz" event={"ID":"2c96886c-3ec5-44c6-9560-8631f7da05c5","Type":"ContainerDied","Data":"0dd28050effa3fc9ac1a911e7b3df908efb5cbbc6daa89d88532cc55b717cbfe"}
Jul 28 01:16:39.694093 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:39.694079    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"
Jul 28 01:16:39.694308 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:39.694146    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd28050effa3fc9ac1a911e7b3df908efb5cbbc6daa89d88532cc55b717cbfe"
Jul 28 01:16:46.263093 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.263058    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"]
Jul 28 01:16:46.263601 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.263313    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" podUID="5bd41250-7ad4-4d75-91d7-200c62bab5ab" containerName="maas-api" containerID="cri-o://e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d" gracePeriod=30
Jul 28 01:16:46.529330 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.529303    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:46.624680 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.624648    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhcnx\" (UniqueName: \"kubernetes.io/projected/5bd41250-7ad4-4d75-91d7-200c62bab5ab-kube-api-access-bhcnx\") pod \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") "
Jul 28 01:16:46.624879 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.624687    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls\") pod \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") "
Jul 28 01:16:46.624879 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.624756    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd41250-7ad4-4d75-91d7-200c62bab5ab-openshift-service-ca\") pod \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\" (UID: \"5bd41250-7ad4-4d75-91d7-200c62bab5ab\") "
Jul 28 01:16:46.625208 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.625178    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd41250-7ad4-4d75-91d7-200c62bab5ab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bd41250-7ad4-4d75-91d7-200c62bab5ab" (UID: "5bd41250-7ad4-4d75-91d7-200c62bab5ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:46.626891 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.626865    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bd41250-7ad4-4d75-91d7-200c62bab5ab" (UID: "5bd41250-7ad4-4d75-91d7-200c62bab5ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:16:46.627003 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.626960    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd41250-7ad4-4d75-91d7-200c62bab5ab-kube-api-access-bhcnx" (OuterVolumeSpecName: "kube-api-access-bhcnx") pod "5bd41250-7ad4-4d75-91d7-200c62bab5ab" (UID: "5bd41250-7ad4-4d75-91d7-200c62bab5ab"). InnerVolumeSpecName "kube-api-access-bhcnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:46.726027 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.725992    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd41250-7ad4-4d75-91d7-200c62bab5ab-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.726027 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.726021    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcnx\" (UniqueName: \"kubernetes.io/projected/5bd41250-7ad4-4d75-91d7-200c62bab5ab-kube-api-access-bhcnx\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.726027 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.726034    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd41250-7ad4-4d75-91d7-200c62bab5ab-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.730143 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.730110    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bd41250-7ad4-4d75-91d7-200c62bab5ab" containerID="e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d" exitCode=0
Jul 28 01:16:46.730282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.730161    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"
Jul 28 01:16:46.730282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.730191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" event={"ID":"5bd41250-7ad4-4d75-91d7-200c62bab5ab","Type":"ContainerDied","Data":"e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d"}
Jul 28 01:16:46.730282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.730274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt" event={"ID":"5bd41250-7ad4-4d75-91d7-200c62bab5ab","Type":"ContainerDied","Data":"3e7ecbc8b36aea5563e24fc1ae83c5e193677378ceac287a15f1b8a323120366"}
Jul 28 01:16:46.730397 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.730298    2564 scope.go:117] "RemoveContainer" containerID="e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d"
Jul 28 01:16:46.739442 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.739421    2564 scope.go:117] "RemoveContainer" containerID="e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d"
Jul 28 01:16:46.739685 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:16:46.739666    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d\": container with ID starting with e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d not found: ID does not exist" containerID="e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d"
Jul 28 01:16:46.739901 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.739695    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d"} err="failed to get container status \"e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d\": rpc error: code = NotFound desc = could not find container \"e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d\": container with ID starting with e43c25cb81651ffcb86c77132df093b065676e2a4720adb26768ad5b1d5a8e4d not found: ID does not exist"
Jul 28 01:16:46.752316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.752279    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"]
Jul 28 01:16:46.755783 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.755749    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6442e46-5564cc5f46-9v6jt"]
Jul 28 01:16:46.945091 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:46.945053    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd41250-7ad4-4d75-91d7-200c62bab5ab" path="/var/lib/kubelet/pods/5bd41250-7ad4-4d75-91d7-200c62bab5ab/volumes"
Jul 28 01:16:49.410256 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:49.410222    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:16:50.777007 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:50.776974    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"]
Jul 28 01:16:50.780126 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:50.780101    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f6442e46-8c6jz"]
Jul 28 01:16:50.904788 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:50.904756    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"]
Jul 28 01:16:50.905057 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:50.905032    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy" containerID="cri-o://79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e" gracePeriod=30
Jul 28 01:16:50.945015 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:50.944987    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c96886c-3ec5-44c6-9560-8631f7da05c5" path="/var/lib/kubelet/pods/2c96886c-3ec5-44c6-9560-8631f7da05c5/volumes"
Jul 28 01:16:56.164363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.164297    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:56.206761 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-data\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.206963 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgb5\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-kube-api-access-kfgb5\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.206963 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206811    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-credential-socket\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.206963 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206849    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-certs\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.206963 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206909    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-envoy\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.206963 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-socket\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.207235 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.206980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d022b15-a124-4332-94d7-72a57de38eae-istio-podinfo\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.207235 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207009    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d022b15-a124-4332-94d7-72a57de38eae-istiod-ca-cert\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.207235 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207044    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-istio-token\") pod \"4d022b15-a124-4332-94d7-72a57de38eae\" (UID: \"4d022b15-a124-4332-94d7-72a57de38eae\") "
Jul 28 01:16:56.207235 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207180    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:16:56.207427 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207364    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.207490 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:16:56.207548 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207496    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d022b15-a124-4332-94d7-72a57de38eae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:56.207548 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207506    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:16:56.207836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.207803    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:16:56.209251 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.209225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-kube-api-access-kfgb5" (OuterVolumeSpecName: "kube-api-access-kfgb5") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "kube-api-access-kfgb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:56.209375 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.209274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d022b15-a124-4332-94d7-72a57de38eae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:16:56.209692 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.209670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:56.209793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.209670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d022b15-a124-4332-94d7-72a57de38eae" (UID: "4d022b15-a124-4332-94d7-72a57de38eae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:16:56.308296 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308259    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308296 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308291    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308296 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308300    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d022b15-a124-4332-94d7-72a57de38eae-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308309    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d022b15-a124-4332-94d7-72a57de38eae-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308318    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308325    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308333    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgb5\" (UniqueName: \"kubernetes.io/projected/4d022b15-a124-4332-94d7-72a57de38eae-kube-api-access-kfgb5\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.308578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.308341    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d022b15-a124-4332-94d7-72a57de38eae-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:16:56.786223 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.786189    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d022b15-a124-4332-94d7-72a57de38eae" containerID="79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e" exitCode=0
Jul 28 01:16:56.786422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.786251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"
Jul 28 01:16:56.786422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.786280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" event={"ID":"4d022b15-a124-4332-94d7-72a57de38eae","Type":"ContainerDied","Data":"79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e"}
Jul 28 01:16:56.786422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.786336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc" event={"ID":"4d022b15-a124-4332-94d7-72a57de38eae","Type":"ContainerDied","Data":"1efcf042622a46790575d335046b6cb1a48b4f9259a3469cac2994d7de0125a5"}
Jul 28 01:16:56.786422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.786359    2564 scope.go:117] "RemoveContainer" containerID="79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e"
Jul 28 01:16:56.795616 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.795597    2564 scope.go:117] "RemoveContainer" containerID="ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"
Jul 28 01:16:56.803756 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.803733    2564 scope.go:117] "RemoveContainer" containerID="79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e"
Jul 28 01:16:56.804017 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:16:56.803997    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e\": container with ID starting with 79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e not found: ID does not exist" containerID="79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e"
Jul 28 01:16:56.804072 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.804027    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e"} err="failed to get container status \"79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e\": rpc error: code = NotFound desc = could not find container \"79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e\": container with ID starting with 79a537df62c45c58abc0f6a129d224c41115189b36e40fd5644515308f823d5e not found: ID does not exist"
Jul 28 01:16:56.804072 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.804046    2564 scope.go:117] "RemoveContainer" containerID="ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"
Jul 28 01:16:56.804273 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:16:56.804254    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658\": container with ID starting with ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658 not found: ID does not exist" containerID="ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"
Jul 28 01:16:56.804314 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.804281    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658"} err="failed to get container status \"ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658\": rpc error: code = NotFound desc = could not find container \"ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658\": container with ID starting with ef5f01caba04796701d78aea34744c7735398aee9c544680586916b77bdba658 not found: ID does not exist"
Jul 28 01:16:56.807331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.807307    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"]
Jul 28 01:16:56.811649 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.811628    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f6442e46-openshift-default-c9d85c544-46kcc"]
Jul 28 01:16:56.945156 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:16:56.945123    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d022b15-a124-4332-94d7-72a57de38eae" path="/var/lib/kubelet/pods/4d022b15-a124-4332-94d7-72a57de38eae/volumes"
Jul 28 01:18:15.303080 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:15.303042    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:18:25.917952 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:25.917906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:18:32.709597 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.709562    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"]
Jul 28 01:18:32.710096 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710004    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy"
Jul 28 01:18:32.710096 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710023    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy"
Jul 28 01:18:32.710096 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710046    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd41250-7ad4-4d75-91d7-200c62bab5ab" containerName="maas-api"
Jul 28 01:18:32.710096 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710054    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd41250-7ad4-4d75-91d7-200c62bab5ab" containerName="maas-api"
Jul 28 01:18:32.710096 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710072    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy"
Jul 28 01:18:32.710096 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710080    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy"
Jul 28 01:18:32.710455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710103    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c96886c-3ec5-44c6-9560-8631f7da05c5" containerName="revoke-keys"
Jul 28 01:18:32.710455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710111    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c96886c-3ec5-44c6-9560-8631f7da05c5" containerName="revoke-keys"
Jul 28 01:18:32.710455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710197    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy"
Jul 28 01:18:32.710455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710209    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd41250-7ad4-4d75-91d7-200c62bab5ab" containerName="maas-api"
Jul 28 01:18:32.710455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710218    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c96886c-3ec5-44c6-9560-8631f7da05c5" containerName="revoke-keys"
Jul 28 01:18:32.710455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.710227    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d022b15-a124-4332-94d7-72a57de38eae" containerName="istio-proxy"
Jul 28 01:18:32.713428 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.713412    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.715673 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.715651    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-730aaa5d-openshift-default-dockercfg-lnchc\""
Jul 28 01:18:32.734004 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.733970    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"]
Jul 28 01:18:32.841916 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.841874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-data\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.841922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-credential-socket\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-envoy\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lfc\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-kube-api-access-g4lfc\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-podinfo\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842175    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-socket\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-token\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istiod-ca-cert\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.842319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.842270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-certs\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-socket\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-token\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istiod-ca-cert\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-certs\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-data\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-credential-socket\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-envoy\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lfc\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-kube-api-access-g4lfc\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.943728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-podinfo\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.944053 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-socket\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.944053 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.943915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-certs\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.944152 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.944119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-credential-socket\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.944343 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.944318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-data\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.944987 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.944964    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istiod-ca-cert\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.946464 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.946443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-envoy\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.946767 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.946744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-podinfo\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.951761 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.951741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-token\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:32.952564 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:32.952545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lfc\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-kube-api-access-g4lfc\") pod \"e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") " pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:33.023324 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.023241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:33.166017 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.165867    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"]
Jul 28 01:18:33.169281 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:18:33.169178    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a25106d_4221_4b52_91b7_ee2bd989f6cb.slice/crio-350aee2f2c5565b8474d099a587acbd6e2832f258a9ffa5260aab3fe02720323 WatchSource:0}: Error finding container 350aee2f2c5565b8474d099a587acbd6e2832f258a9ffa5260aab3fe02720323: Status 404 returned error can't find the container with id 350aee2f2c5565b8474d099a587acbd6e2832f258a9ffa5260aab3fe02720323
Jul 28 01:18:33.171832 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.171792    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:18:33.171913 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.171861    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:18:33.171913 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.171895    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:18:33.255009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.254969    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" event={"ID":"5a25106d-4221-4b52-91b7-ee2bd989f6cb","Type":"ContainerStarted","Data":"96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"}
Jul 28 01:18:33.255009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.255037    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" event={"ID":"5a25106d-4221-4b52-91b7-ee2bd989f6cb","Type":"ContainerStarted","Data":"350aee2f2c5565b8474d099a587acbd6e2832f258a9ffa5260aab3fe02720323"}
Jul 28 01:18:33.279340 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:33.279229    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podStartSLOduration=1.279207306 podStartE2EDuration="1.279207306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:18:33.276425515 +0000 UTC m=+1944.831928695" watchObservedRunningTime="2026-07-28 01:18:33.279207306 +0000 UTC m=+1944.834710465"
Jul 28 01:18:34.024046 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:34.024004    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:34.029254 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:34.029224    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:34.260245 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:34.260210    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:34.261124 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:34.261099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:35.298755 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.298718    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:18:35.619102 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.619013    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"]
Jul 28 01:18:35.622702 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.622677    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.624895 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.624862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-730aaa5d-serving-cert\""
Jul 28 01:18:35.625212 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.625191    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:18:35.625323 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.625214    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:18:35.625747 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.625729    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:18:35.633414 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.633370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"]
Jul 28 01:18:35.666577 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.666539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e24c047-32b1-4b66-bb4c-0fe1876f5644-openshift-service-ca\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.666577 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.666579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.666798 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.666607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwc7n\" (UniqueName: \"kubernetes.io/projected/0e24c047-32b1-4b66-bb4c-0fe1876f5644-kube-api-access-xwc7n\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.767165 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.767123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.767370 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.767171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwc7n\" (UniqueName: \"kubernetes.io/projected/0e24c047-32b1-4b66-bb4c-0fe1876f5644-kube-api-access-xwc7n\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.767370 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:18:35.767269    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-serving-cert: secret "maas-api-e2e-mt-730aaa5d-serving-cert" not found
Jul 28 01:18:35.767370 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.767301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e24c047-32b1-4b66-bb4c-0fe1876f5644-openshift-service-ca\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.767370 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:18:35.767332    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls podName:0e24c047-32b1-4b66-bb4c-0fe1876f5644 nodeName:}" failed. No retries permitted until 2026-07-28 01:18:36.267316796 +0000 UTC m=+1947.822819950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls") pod "maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" (UID: "0e24c047-32b1-4b66-bb4c-0fe1876f5644") : secret "maas-api-e2e-mt-730aaa5d-serving-cert" not found
Jul 28 01:18:35.767955 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.767919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e24c047-32b1-4b66-bb4c-0fe1876f5644-openshift-service-ca\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:35.781913 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:35.781881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwc7n\" (UniqueName: \"kubernetes.io/projected/0e24c047-32b1-4b66-bb4c-0fe1876f5644-kube-api-access-xwc7n\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:36.271421 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:36.271383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:36.273805 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:36.273781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls\") pod \"maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:36.537921 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:36.537816    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:36.681144 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:36.681116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"]
Jul 28 01:18:36.683536 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:18:36.683506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e24c047_32b1_4b66_bb4c_0fe1876f5644.slice/crio-8774a0a7fb203229d5aad49986019119ea7e0207aaa2d46db707775334e4b142 WatchSource:0}: Error finding container 8774a0a7fb203229d5aad49986019119ea7e0207aaa2d46db707775334e4b142: Status 404 returned error can't find the container with id 8774a0a7fb203229d5aad49986019119ea7e0207aaa2d46db707775334e4b142
Jul 28 01:18:37.278579 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:37.278543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" event={"ID":"0e24c047-32b1-4b66-bb4c-0fe1876f5644","Type":"ContainerStarted","Data":"8774a0a7fb203229d5aad49986019119ea7e0207aaa2d46db707775334e4b142"}
Jul 28 01:18:38.285629 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:38.285596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" event={"ID":"0e24c047-32b1-4b66-bb4c-0fe1876f5644","Type":"ContainerStarted","Data":"f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c"}
Jul 28 01:18:38.286086 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:38.285708    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:38.302811 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:38.302761    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" podStartSLOduration=2.629669217 podStartE2EDuration="3.30274501s" totalImagesPullingTime="673.075793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:18:38.300869971 +0000 UTC m=+1949.856373280" watchObservedRunningTime="2026-07-28 01:18:38.30274501 +0000 UTC m=+1949.858248211"
Jul 28 01:18:44.295420 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:44.295388    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:46.404107 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:46.404073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:18:46.951578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:46.951214    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"]
Jul 28 01:18:46.956167 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:46.956138    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:46.958470 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:46.958440    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:18:46.960395 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:46.960370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"]
Jul 28 01:18:47.072364 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:47.072319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d43bdb-93d8-439d-aae0-371a64c40266-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4\" (UID: \"f1d43bdb-93d8-439d-aae0-371a64c40266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:47.173002 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:47.172958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d43bdb-93d8-439d-aae0-371a64c40266-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4\" (UID: \"f1d43bdb-93d8-439d-aae0-371a64c40266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:47.173860 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:47.173833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d43bdb-93d8-439d-aae0-371a64c40266-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4\" (UID: \"f1d43bdb-93d8-439d-aae0-371a64c40266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:47.271115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:47.271025    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:47.423847 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:47.423817    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"]
Jul 28 01:18:47.426649 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:18:47.426615    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d43bdb_93d8_439d_aae0_371a64c40266.slice/crio-efe1ca250c0d260e7ce4349b898bf20e6dd0b16e90688cdb85a44ec0f28b06e8 WatchSource:0}: Error finding container efe1ca250c0d260e7ce4349b898bf20e6dd0b16e90688cdb85a44ec0f28b06e8: Status 404 returned error can't find the container with id efe1ca250c0d260e7ce4349b898bf20e6dd0b16e90688cdb85a44ec0f28b06e8
Jul 28 01:18:48.336762 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:48.336722    2564 generic.go:356] "Generic (PLEG): container finished" podID="f1d43bdb-93d8-439d-aae0-371a64c40266" containerID="1244ad47ff2cf3f11b2ec4f19a85319d00f18e7d461deba0b5a7274c1ff32d01" exitCode=0
Jul 28 01:18:48.336988 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:48.336791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4" event={"ID":"f1d43bdb-93d8-439d-aae0-371a64c40266","Type":"ContainerDied","Data":"1244ad47ff2cf3f11b2ec4f19a85319d00f18e7d461deba0b5a7274c1ff32d01"}
Jul 28 01:18:48.336988 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:48.336845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4" event={"ID":"f1d43bdb-93d8-439d-aae0-371a64c40266","Type":"ContainerStarted","Data":"efe1ca250c0d260e7ce4349b898bf20e6dd0b16e90688cdb85a44ec0f28b06e8"}
Jul 28 01:18:49.261783 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.261725    2564 patch_prober.go:28] interesting pod/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 28 01:18:49.262247 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.261806    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 28 01:18:49.345014 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.344980    2564 generic.go:356] "Generic (PLEG): container finished" podID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerID="96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862" exitCode=0
Jul 28 01:18:49.345187 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.345057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" event={"ID":"5a25106d-4221-4b52-91b7-ee2bd989f6cb","Type":"ContainerDied","Data":"96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"}
Jul 28 01:18:49.346051 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.345541    2564 scope.go:117] "RemoveContainer" containerID="96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"
Jul 28 01:18:49.346679 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.346639    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:18:49.346786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.346717    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:18:49.346786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.346766    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:18:49.511354 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.511325    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:49.596916 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.596808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d43bdb-93d8-439d-aae0-371a64c40266-maas-api-service-ca\") pod \"f1d43bdb-93d8-439d-aae0-371a64c40266\" (UID: \"f1d43bdb-93d8-439d-aae0-371a64c40266\") "
Jul 28 01:18:49.597181 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.597158    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d43bdb-93d8-439d-aae0-371a64c40266-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1d43bdb-93d8-439d-aae0-371a64c40266" (UID: "f1d43bdb-93d8-439d-aae0-371a64c40266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:18:49.698104 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:49.698058    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d43bdb-93d8-439d-aae0-371a64c40266-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:18:50.352342 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:50.352311    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"
Jul 28 01:18:50.352829 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:50.352355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4" event={"ID":"f1d43bdb-93d8-439d-aae0-371a64c40266","Type":"ContainerDied","Data":"efe1ca250c0d260e7ce4349b898bf20e6dd0b16e90688cdb85a44ec0f28b06e8"}
Jul 28 01:18:50.352829 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:50.352408    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efe1ca250c0d260e7ce4349b898bf20e6dd0b16e90688cdb85a44ec0f28b06e8"
Jul 28 01:18:50.354303 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:50.354271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" event={"ID":"5a25106d-4221-4b52-91b7-ee2bd989f6cb","Type":"ContainerStarted","Data":"727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624"}
Jul 28 01:18:51.024326 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:51.024289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:51.024550 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:51.024526    2564 patch_prober.go:28] interesting pod/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 28 01:18:51.024609 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:51.024576    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 28 01:18:52.024165 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:52.024127    2564 patch_prober.go:28] interesting pod/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 28 01:18:52.024556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:52.024184    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 28 01:18:53.024070 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:53.024031    2564 patch_prober.go:28] interesting pod/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 28 01:18:53.024256 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:53.024084    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 28 01:18:54.023764 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:54.023729    2564 patch_prober.go:28] interesting pod/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 28 01:18:54.023971 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:54.023785    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 28 01:18:55.028971 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:55.028918    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:55.029352 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:55.029314    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:55.030181 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:55.030159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:18:57.508037 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.507997    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"]
Jul 28 01:18:57.508621 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.508244    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" podUID="0e24c047-32b1-4b66-bb4c-0fe1876f5644" containerName="maas-api" containerID="cri-o://f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c" gracePeriod=30
Jul 28 01:18:57.760871 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.760787    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:57.874883 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.874848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e24c047-32b1-4b66-bb4c-0fe1876f5644-openshift-service-ca\") pod \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") "
Jul 28 01:18:57.875085 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.874899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls\") pod \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") "
Jul 28 01:18:57.875085 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.875012    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwc7n\" (UniqueName: \"kubernetes.io/projected/0e24c047-32b1-4b66-bb4c-0fe1876f5644-kube-api-access-xwc7n\") pod \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\" (UID: \"0e24c047-32b1-4b66-bb4c-0fe1876f5644\") "
Jul 28 01:18:57.875323 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.875289    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e24c047-32b1-4b66-bb4c-0fe1876f5644-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e24c047-32b1-4b66-bb4c-0fe1876f5644" (UID: "0e24c047-32b1-4b66-bb4c-0fe1876f5644"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:18:57.876970 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.876946    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e24c047-32b1-4b66-bb4c-0fe1876f5644" (UID: "0e24c047-32b1-4b66-bb4c-0fe1876f5644"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:18:57.877074 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.877046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e24c047-32b1-4b66-bb4c-0fe1876f5644-kube-api-access-xwc7n" (OuterVolumeSpecName: "kube-api-access-xwc7n") pod "0e24c047-32b1-4b66-bb4c-0fe1876f5644" (UID: "0e24c047-32b1-4b66-bb4c-0fe1876f5644"). InnerVolumeSpecName "kube-api-access-xwc7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:18:57.976295 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.976260    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e24c047-32b1-4b66-bb4c-0fe1876f5644-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:18:57.976295 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.976288    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwc7n\" (UniqueName: \"kubernetes.io/projected/0e24c047-32b1-4b66-bb4c-0fe1876f5644-kube-api-access-xwc7n\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:18:57.976295 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:57.976297    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e24c047-32b1-4b66-bb4c-0fe1876f5644-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:18:58.400011 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.399974    2564 generic.go:356] "Generic (PLEG): container finished" podID="0e24c047-32b1-4b66-bb4c-0fe1876f5644" containerID="f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c" exitCode=0
Jul 28 01:18:58.400196 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.400031    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"
Jul 28 01:18:58.400196 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.400061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" event={"ID":"0e24c047-32b1-4b66-bb4c-0fe1876f5644","Type":"ContainerDied","Data":"f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c"}
Jul 28 01:18:58.400196 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.400145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk" event={"ID":"0e24c047-32b1-4b66-bb4c-0fe1876f5644","Type":"ContainerDied","Data":"8774a0a7fb203229d5aad49986019119ea7e0207aaa2d46db707775334e4b142"}
Jul 28 01:18:58.400196 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.400169    2564 scope.go:117] "RemoveContainer" containerID="f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c"
Jul 28 01:18:58.409478 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.409458    2564 scope.go:117] "RemoveContainer" containerID="f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c"
Jul 28 01:18:58.409782 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:18:58.409746    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c\": container with ID starting with f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c not found: ID does not exist" containerID="f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c"
Jul 28 01:18:58.409823 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.409791    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c"} err="failed to get container status \"f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c\": rpc error: code = NotFound desc = could not find container \"f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c\": container with ID starting with f6345e7b49f44009fdabce436801310c8d45b46fb72e5552ce168a90709cae8c not found: ID does not exist"
Jul 28 01:18:58.423838 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.423809    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"]
Jul 28 01:18:58.427948 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.427915    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-730aaa5d-7cdfcb466f-qqflk"]
Jul 28 01:18:58.945395 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:18:58.945365    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e24c047-32b1-4b66-bb4c-0fe1876f5644" path="/var/lib/kubelet/pods/0e24c047-32b1-4b66-bb4c-0fe1876f5644/volumes"
Jul 28 01:19:00.608115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:00.608081    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:19:02.017569 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:02.017471    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"]
Jul 28 01:19:02.019552 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:02.019528    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-730aaa5d-z9wv4"]
Jul 28 01:19:02.944523 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:02.944482    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1d43bdb-93d8-439d-aae0-371a64c40266" path="/var/lib/kubelet/pods/f1d43bdb-93d8-439d-aae0-371a64c40266/volumes"
Jul 28 01:19:08.502977 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:08.502921    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"]
Jul 28 01:19:08.503409 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:08.503176    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" containerID="cri-o://727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624" gracePeriod=30
Jul 28 01:19:10.032154 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:10.032112    2564 patch_prober.go:28] interesting pod/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 01:19:10.032541 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:10.032191    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:19:13.755704 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.755680    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:19:13.811044 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.810924    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istiod-ca-cert\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811044 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.810986    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4lfc\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-kube-api-access-g4lfc\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811044 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811009    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-socket\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811044 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811037    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-envoy\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-certs\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811082    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-token\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811123    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-credential-socket\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-podinfo\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811159    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-data\") pod \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\" (UID: \"5a25106d-4221-4b52-91b7-ee2bd989f6cb\") "
Jul 28 01:19:13.811373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811346    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:19:13.811667 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811362    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:19:13.811667 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811405    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:19:13.811776 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.811704    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:19:13.812050 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.812029    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:19:13.813200 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.813177    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-kube-api-access-g4lfc" (OuterVolumeSpecName: "kube-api-access-g4lfc") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "kube-api-access-g4lfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:19:13.813379 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.813358    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:19:13.813549 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.813531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:19:13.813626 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.813609    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a25106d-4221-4b52-91b7-ee2bd989f6cb" (UID: "5a25106d-4221-4b52-91b7-ee2bd989f6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:19:13.912316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912279    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912308    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912316    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912326    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912334    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lfc\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-kube-api-access-g4lfc\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912343    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912351    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912359    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a25106d-4221-4b52-91b7-ee2bd989f6cb-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:13.912584 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:13.912366    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a25106d-4221-4b52-91b7-ee2bd989f6cb-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:19:14.482207 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.482174    2564 generic.go:356] "Generic (PLEG): container finished" podID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerID="727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624" exitCode=0
Jul 28 01:19:14.482364 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.482218    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"
Jul 28 01:19:14.482364 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.482259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" event={"ID":"5a25106d-4221-4b52-91b7-ee2bd989f6cb","Type":"ContainerDied","Data":"727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624"}
Jul 28 01:19:14.482364 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.482314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd" event={"ID":"5a25106d-4221-4b52-91b7-ee2bd989f6cb","Type":"ContainerDied","Data":"350aee2f2c5565b8474d099a587acbd6e2832f258a9ffa5260aab3fe02720323"}
Jul 28 01:19:14.482364 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.482337    2564 scope.go:117] "RemoveContainer" containerID="727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624"
Jul 28 01:19:14.491518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.491502    2564 scope.go:117] "RemoveContainer" containerID="96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"
Jul 28 01:19:14.501090 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.501070    2564 scope.go:117] "RemoveContainer" containerID="727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624"
Jul 28 01:19:14.501583 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:19:14.501565    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624\": container with ID starting with 727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624 not found: ID does not exist" containerID="727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624"
Jul 28 01:19:14.501638 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.501593    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624"} err="failed to get container status \"727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624\": rpc error: code = NotFound desc = could not find container \"727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624\": container with ID starting with 727f1ad6337ba3836acdeb6e3bb1838f9928ac5b849f9f7290cbd7c813e01624 not found: ID does not exist"
Jul 28 01:19:14.501638 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.501613    2564 scope.go:117] "RemoveContainer" containerID="96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"
Jul 28 01:19:14.501930 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:19:14.501906    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862\": container with ID starting with 96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862 not found: ID does not exist" containerID="96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"
Jul 28 01:19:14.502062 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.501953    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862"} err="failed to get container status \"96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862\": rpc error: code = NotFound desc = could not find container \"96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862\": container with ID starting with 96cf4156436d87431f414b79dba836a819b36eb6d1ee3f26612c1a7f2cdab862 not found: ID does not exist"
Jul 28 01:19:14.506308 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.506284    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"]
Jul 28 01:19:14.509064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.509045    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-730aaa5d-openshift-default-84c5786bfb-l4fmd"]
Jul 28 01:19:14.945378 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:14.945347    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" path="/var/lib/kubelet/pods/5a25106d-4221-4b52-91b7-ee2bd989f6cb/volumes"
Jul 28 01:19:28.520280 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520239    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"]
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520606    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d43bdb-93d8-439d-aae0-371a64c40266" containerName="revoke-keys"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520620    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d43bdb-93d8-439d-aae0-371a64c40266" containerName="revoke-keys"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520629    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520634    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520645    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e24c047-32b1-4b66-bb4c-0fe1876f5644" containerName="maas-api"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520653    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e24c047-32b1-4b66-bb4c-0fe1876f5644" containerName="maas-api"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520669    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520674    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520732    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520742    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e24c047-32b1-4b66-bb4c-0fe1876f5644" containerName="maas-api"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520748    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a25106d-4221-4b52-91b7-ee2bd989f6cb" containerName="istio-proxy"
Jul 28 01:19:28.520784 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.520754    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d43bdb-93d8-439d-aae0-371a64c40266" containerName="revoke-keys"
Jul 28 01:19:28.523948 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.523915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.526394 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.526371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7ce60076-openshift-default-dockercfg-bcmqp\""
Jul 28 01:19:28.535091 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.535062    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"]
Jul 28 01:19:28.644686 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.644651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-credential-socket\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.644686 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.644689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cacfa-3690-473c-b8b3-d676f461d103-istiod-ca-cert\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.644898 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.644709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7rg\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-kube-api-access-5z7rg\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.644898 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.644758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cacfa-3690-473c-b8b3-d676f461d103-istio-podinfo\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.645002 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.644900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-envoy\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.645002 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.644926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-istio-token\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.645066 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.645004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-socket\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.645066 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.645035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-data\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.645066 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.645052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-certs\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746223 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-data\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746223 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-certs\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746279    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-credential-socket\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cacfa-3690-473c-b8b3-d676f461d103-istiod-ca-cert\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7rg\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-kube-api-access-5z7rg\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cacfa-3690-473c-b8b3-d676f461d103-istio-podinfo\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-envoy\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746488 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-istio-token\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746806 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-socket\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746806 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-certs\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746806 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-data\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.746806 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-credential-socket\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.747037 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.746838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-socket\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.747198 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.747180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cacfa-3690-473c-b8b3-d676f461d103-istiod-ca-cert\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.748816 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.748793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-envoy\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.749063 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.749044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cacfa-3690-473c-b8b3-d676f461d103-istio-podinfo\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.763525 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.763483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-istio-token\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.764056 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.764034    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7rg\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-kube-api-access-5z7rg\") pod \"e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") " pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.834873 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.834777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:28.979551 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.979521    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"]
Jul 28 01:19:28.982302 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:19:28.982274    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344cacfa_3690_473c_b8b3_d676f461d103.slice/crio-9cdbb4034e13e9471e2e368daec5cdedc00a4befcf66cbe5168974f4fc18b08e WatchSource:0}: Error finding container 9cdbb4034e13e9471e2e368daec5cdedc00a4befcf66cbe5168974f4fc18b08e: Status 404 returned error can't find the container with id 9cdbb4034e13e9471e2e368daec5cdedc00a4befcf66cbe5168974f4fc18b08e
Jul 28 01:19:28.984371 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.984332    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:19:28.984473 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.984405    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:19:28.984473 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:28.984431    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:19:29.560457 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:29.560419    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" event={"ID":"344cacfa-3690-473c-b8b3-d676f461d103","Type":"ContainerStarted","Data":"12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"}
Jul 28 01:19:29.560988 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:29.560477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" event={"ID":"344cacfa-3690-473c-b8b3-d676f461d103","Type":"ContainerStarted","Data":"9cdbb4034e13e9471e2e368daec5cdedc00a4befcf66cbe5168974f4fc18b08e"}
Jul 28 01:19:29.580206 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:29.580150    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podStartSLOduration=1.5801352739999999 podStartE2EDuration="1.580135274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:19:29.577925224 +0000 UTC m=+2001.133428394" watchObservedRunningTime="2026-07-28 01:19:29.580135274 +0000 UTC m=+2001.135638434"
Jul 28 01:19:29.835922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:29.835834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:30.840349 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:30.840316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:31.539687 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.539651    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"]
Jul 28 01:19:31.543311 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.543281    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.545695 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.545665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ce60076-serving-cert\""
Jul 28 01:19:31.545831 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.545743    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:19:31.546522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.546498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:19:31.546522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.546514    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:19:31.553175 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.553150    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"]
Jul 28 01:19:31.571620 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.571581    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:31.572432 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.572403    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:31.678944 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.678897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mchp\" (UniqueName: \"kubernetes.io/projected/8820f73c-64b5-48a3-aa83-e178a78f1564-kube-api-access-6mchp\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.679153 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.678967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8820f73c-64b5-48a3-aa83-e178a78f1564-openshift-service-ca\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.679153 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.679108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.780315 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.780273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.780482 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.780365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mchp\" (UniqueName: \"kubernetes.io/projected/8820f73c-64b5-48a3-aa83-e178a78f1564-kube-api-access-6mchp\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.780482 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.780424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8820f73c-64b5-48a3-aa83-e178a78f1564-openshift-service-ca\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.780482 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:19:31.780435    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-serving-cert: secret "maas-api-e2e-mt-7ce60076-serving-cert" not found
Jul 28 01:19:31.780628 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:19:31.780498    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls podName:8820f73c-64b5-48a3-aa83-e178a78f1564 nodeName:}" failed. No retries permitted until 2026-07-28 01:19:32.28047764 +0000 UTC m=+2003.835980794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls") pod "maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" (UID: "8820f73c-64b5-48a3-aa83-e178a78f1564") : secret "maas-api-e2e-mt-7ce60076-serving-cert" not found
Jul 28 01:19:31.781091 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.781072    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8820f73c-64b5-48a3-aa83-e178a78f1564-openshift-service-ca\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:31.794054 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:31.793981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mchp\" (UniqueName: \"kubernetes.io/projected/8820f73c-64b5-48a3-aa83-e178a78f1564-kube-api-access-6mchp\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:32.286689 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:32.286634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:32.289442 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:32.289408    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls\") pod \"maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:32.455683 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:32.455636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:32.610621 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:32.610597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"]
Jul 28 01:19:32.612665 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:19:32.612631    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8820f73c_64b5_48a3_aa83_e178a78f1564.slice/crio-c1708ca005b8947bae5df36ad0a9161e7b4f668f7aa76ee0ad7a382717f3b25e WatchSource:0}: Error finding container c1708ca005b8947bae5df36ad0a9161e7b4f668f7aa76ee0ad7a382717f3b25e: Status 404 returned error can't find the container with id c1708ca005b8947bae5df36ad0a9161e7b4f668f7aa76ee0ad7a382717f3b25e
Jul 28 01:19:32.614171 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:32.614149    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:19:33.584920 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:33.584879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" event={"ID":"8820f73c-64b5-48a3-aa83-e178a78f1564","Type":"ContainerStarted","Data":"7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0"}
Jul 28 01:19:33.585361 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:33.585011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" event={"ID":"8820f73c-64b5-48a3-aa83-e178a78f1564","Type":"ContainerStarted","Data":"c1708ca005b8947bae5df36ad0a9161e7b4f668f7aa76ee0ad7a382717f3b25e"}
Jul 28 01:19:33.585361 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:33.585230    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:33.601463 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:33.601408    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" podStartSLOduration=2.1177593 podStartE2EDuration="2.601393553s" totalImagesPullingTime="483.634253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:19:33.599267945 +0000 UTC m=+2005.154771105" watchObservedRunningTime="2026-07-28 01:19:33.601393553 +0000 UTC m=+2005.156896713"
Jul 28 01:19:37.712410 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:37.712378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:19:39.596216 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:39.596187    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:19:46.573921 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.573883    2564 patch_prober.go:28] interesting pod/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 28 01:19:46.574403 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.573967    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 28 01:19:46.655095 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.655059    2564 generic.go:356] "Generic (PLEG): container finished" podID="344cacfa-3690-473c-b8b3-d676f461d103" containerID="12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8" exitCode=0
Jul 28 01:19:46.655259 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.655127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" event={"ID":"344cacfa-3690-473c-b8b3-d676f461d103","Type":"ContainerDied","Data":"12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"}
Jul 28 01:19:46.655605 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.655588    2564 scope.go:117] "RemoveContainer" containerID="12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"
Jul 28 01:19:46.656682 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.656655    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:19:46.656738 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.656723    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:19:46.656777 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:46.656753    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:19:47.663544 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:47.663474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" event={"ID":"344cacfa-3690-473c-b8b3-d676f461d103","Type":"ContainerStarted","Data":"8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96"}
Jul 28 01:19:47.835704 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:47.835663    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:47.835992 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:47.835964    2564 patch_prober.go:28] interesting pod/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 28 01:19:47.836049 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:47.836022    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 28 01:19:48.835554 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:48.835509    2564 patch_prober.go:28] interesting pod/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 28 01:19:48.836035 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:48.835585    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 28 01:19:49.835909 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:49.835870    2564 patch_prober.go:28] interesting pod/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 28 01:19:49.836364 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:49.835971    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 28 01:19:50.835439 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:50.835397    2564 patch_prober.go:28] interesting pod/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 28 01:19:50.835635 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:50.835453    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 28 01:19:51.839568 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:51.839535    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:51.839975 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:51.839875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:51.840519 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:51.840501    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:19:52.110442 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:19:52.110354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:20:06.141335 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:06.141289    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:20:10.270920 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.270880    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"]
Jul 28 01:20:10.275593 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.275563    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:10.277947 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.277918    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:20:10.283828 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.283102    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"]
Jul 28 01:20:10.315413 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.315376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c55d25-872c-4b3a-bebe-23c9ade1e373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ce60076-gnchv\" (UID: \"34c55d25-872c-4b3a-bebe-23c9ade1e373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:10.415897 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.415852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c55d25-872c-4b3a-bebe-23c9ade1e373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ce60076-gnchv\" (UID: \"34c55d25-872c-4b3a-bebe-23c9ade1e373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:10.416495 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.416474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c55d25-872c-4b3a-bebe-23c9ade1e373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ce60076-gnchv\" (UID: \"34c55d25-872c-4b3a-bebe-23c9ade1e373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:10.587609 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.587515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:10.729885 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.729852    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"]
Jul 28 01:20:10.730684 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:20:10.730659    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c55d25_872c_4b3a_bebe_23c9ade1e373.slice/crio-025bfc083438b51005170c3e917897145fe98a78aaf250330249736683a6b80d WatchSource:0}: Error finding container 025bfc083438b51005170c3e917897145fe98a78aaf250330249736683a6b80d: Status 404 returned error can't find the container with id 025bfc083438b51005170c3e917897145fe98a78aaf250330249736683a6b80d
Jul 28 01:20:10.789410 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:10.789375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv" event={"ID":"34c55d25-872c-4b3a-bebe-23c9ade1e373","Type":"ContainerStarted","Data":"025bfc083438b51005170c3e917897145fe98a78aaf250330249736683a6b80d"}
Jul 28 01:20:11.795999 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:11.795872    2564 generic.go:356] "Generic (PLEG): container finished" podID="34c55d25-872c-4b3a-bebe-23c9ade1e373" containerID="af4ffe468508ffaa1f87d9879e48cab1f8de3f96f553e62e91187ad8117e902a" exitCode=0
Jul 28 01:20:11.795999 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:11.795972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv" event={"ID":"34c55d25-872c-4b3a-bebe-23c9ade1e373","Type":"ContainerDied","Data":"af4ffe468508ffaa1f87d9879e48cab1f8de3f96f553e62e91187ad8117e902a"}
Jul 28 01:20:12.941323 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:12.941290    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:13.037406 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:13.037357    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c55d25-872c-4b3a-bebe-23c9ade1e373-maas-api-service-ca\") pod \"34c55d25-872c-4b3a-bebe-23c9ade1e373\" (UID: \"34c55d25-872c-4b3a-bebe-23c9ade1e373\") "
Jul 28 01:20:13.037750 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:13.037724    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c55d25-872c-4b3a-bebe-23c9ade1e373-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34c55d25-872c-4b3a-bebe-23c9ade1e373" (UID: "34c55d25-872c-4b3a-bebe-23c9ade1e373"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:20:13.138706 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:13.138621    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c55d25-872c-4b3a-bebe-23c9ade1e373-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:20:13.807271 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:13.807244    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"
Jul 28 01:20:13.807452 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:13.807270    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv" event={"ID":"34c55d25-872c-4b3a-bebe-23c9ade1e373","Type":"ContainerDied","Data":"025bfc083438b51005170c3e917897145fe98a78aaf250330249736683a6b80d"}
Jul 28 01:20:13.807452 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:13.807319    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025bfc083438b51005170c3e917897145fe98a78aaf250330249736683a6b80d"
Jul 28 01:20:24.710415 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:24.710379    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:20:25.988110 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:25.988070    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"]
Jul 28 01:20:25.988572 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:25.988387    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" podUID="8820f73c-64b5-48a3-aa83-e178a78f1564" containerName="maas-api" containerID="cri-o://7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0" gracePeriod=30
Jul 28 01:20:26.246185 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.246123    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:20:26.356788 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.356751    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mchp\" (UniqueName: \"kubernetes.io/projected/8820f73c-64b5-48a3-aa83-e178a78f1564-kube-api-access-6mchp\") pod \"8820f73c-64b5-48a3-aa83-e178a78f1564\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") "
Jul 28 01:20:26.357017 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.356796    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls\") pod \"8820f73c-64b5-48a3-aa83-e178a78f1564\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") "
Jul 28 01:20:26.357017 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.356848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8820f73c-64b5-48a3-aa83-e178a78f1564-openshift-service-ca\") pod \"8820f73c-64b5-48a3-aa83-e178a78f1564\" (UID: \"8820f73c-64b5-48a3-aa83-e178a78f1564\") "
Jul 28 01:20:26.357299 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.357274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8820f73c-64b5-48a3-aa83-e178a78f1564-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8820f73c-64b5-48a3-aa83-e178a78f1564" (UID: "8820f73c-64b5-48a3-aa83-e178a78f1564"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:20:26.358910 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.358886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8820f73c-64b5-48a3-aa83-e178a78f1564" (UID: "8820f73c-64b5-48a3-aa83-e178a78f1564"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:20:26.359049 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.358979    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8820f73c-64b5-48a3-aa83-e178a78f1564-kube-api-access-6mchp" (OuterVolumeSpecName: "kube-api-access-6mchp") pod "8820f73c-64b5-48a3-aa83-e178a78f1564" (UID: "8820f73c-64b5-48a3-aa83-e178a78f1564"). InnerVolumeSpecName "kube-api-access-6mchp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:20:26.458510 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.458475    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mchp\" (UniqueName: \"kubernetes.io/projected/8820f73c-64b5-48a3-aa83-e178a78f1564-kube-api-access-6mchp\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.458510 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.458506    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8820f73c-64b5-48a3-aa83-e178a78f1564-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.458510 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.458515    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8820f73c-64b5-48a3-aa83-e178a78f1564-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.882821 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.882781    2564 generic.go:356] "Generic (PLEG): container finished" podID="8820f73c-64b5-48a3-aa83-e178a78f1564" containerID="7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0" exitCode=0
Jul 28 01:20:26.883028 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.882828    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"
Jul 28 01:20:26.883028 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.882828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" event={"ID":"8820f73c-64b5-48a3-aa83-e178a78f1564","Type":"ContainerDied","Data":"7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0"}
Jul 28 01:20:26.883028 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.882979    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2" event={"ID":"8820f73c-64b5-48a3-aa83-e178a78f1564","Type":"ContainerDied","Data":"c1708ca005b8947bae5df36ad0a9161e7b4f668f7aa76ee0ad7a382717f3b25e"}
Jul 28 01:20:26.883028 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.883013    2564 scope.go:117] "RemoveContainer" containerID="7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0"
Jul 28 01:20:26.892567 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.892551    2564 scope.go:117] "RemoveContainer" containerID="7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0"
Jul 28 01:20:26.892815 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:20:26.892796    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0\": container with ID starting with 7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0 not found: ID does not exist" containerID="7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0"
Jul 28 01:20:26.892859 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.892824    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0"} err="failed to get container status \"7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0\": rpc error: code = NotFound desc = could not find container \"7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0\": container with ID starting with 7dd30d41e6024c7083361a3c2581574abf979e5901c5d8e47d8a620e1d0b49c0 not found: ID does not exist"
Jul 28 01:20:26.904774 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.904745    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"]
Jul 28 01:20:26.908038 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.908017    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ce60076-84467dbbc-dmhs2"]
Jul 28 01:20:26.945473 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:26.945449    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8820f73c-64b5-48a3-aa83-e178a78f1564" path="/var/lib/kubelet/pods/8820f73c-64b5-48a3-aa83-e178a78f1564/volumes"
Jul 28 01:20:30.383623 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:30.383583    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"]
Jul 28 01:20:30.389952 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:30.389904    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ce60076-gnchv"]
Jul 28 01:20:30.945025 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:30.944993    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c55d25-872c-4b3a-bebe-23c9ade1e373" path="/var/lib/kubelet/pods/34c55d25-872c-4b3a-bebe-23c9ade1e373/volumes"
Jul 28 01:20:35.330269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:20:35.330233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:21:09.029443 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:09.029416    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:21:09.037482 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:09.037455    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:21:09.101359 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:09.101333    2564 scope.go:117] "RemoveContainer" containerID="0ec13282d9dc2e9dab54a852ebe7d7f99f4d8741bbda847d4221947a6f87fe5d"
Jul 28 01:21:09.109501 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:09.109485    2564 scope.go:117] "RemoveContainer" containerID="a2533e6d58ad2c7d9f9c728c7d182be8ffc83e0121e1ca68cf1e42dc45386dd8"
Jul 28 01:21:09.117059 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:09.117040    2564 scope.go:117] "RemoveContainer" containerID="57c04ffc729429a61d8b04d81ed5d8cfd170b202458dc99e0fed0af8eeb9cebb"
Jul 28 01:21:28.928400 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:28.928360    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"]
Jul 28 01:21:28.928993 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:28.928582    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy" containerID="cri-o://8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96" gracePeriod=30
Jul 28 01:21:33.002314 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:33.002278    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:21:34.191845 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.191777    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:21:34.216447 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.216418    2564 generic.go:356] "Generic (PLEG): container finished" podID="344cacfa-3690-473c-b8b3-d676f461d103" containerID="8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96" exitCode=0
Jul 28 01:21:34.216607 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.216469    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"
Jul 28 01:21:34.216607 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.216499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" event={"ID":"344cacfa-3690-473c-b8b3-d676f461d103","Type":"ContainerDied","Data":"8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96"}
Jul 28 01:21:34.216607 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.216593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4" event={"ID":"344cacfa-3690-473c-b8b3-d676f461d103","Type":"ContainerDied","Data":"9cdbb4034e13e9471e2e368daec5cdedc00a4befcf66cbe5168974f4fc18b08e"}
Jul 28 01:21:34.216717 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.216624    2564 scope.go:117] "RemoveContainer" containerID="8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96"
Jul 28 01:21:34.226482 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.226461    2564 scope.go:117] "RemoveContainer" containerID="12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"
Jul 28 01:21:34.234752 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.234730    2564 scope.go:117] "RemoveContainer" containerID="8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96"
Jul 28 01:21:34.235085 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:21:34.235063    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96\": container with ID starting with 8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96 not found: ID does not exist" containerID="8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96"
Jul 28 01:21:34.235167 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.235097    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96"} err="failed to get container status \"8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96\": rpc error: code = NotFound desc = could not find container \"8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96\": container with ID starting with 8f7685ea2b369802123cddbbf30b7038aa8bc3957d5ba27f6514db8804e33b96 not found: ID does not exist"
Jul 28 01:21:34.235167 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.235116    2564 scope.go:117] "RemoveContainer" containerID="12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"
Jul 28 01:21:34.235368 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:21:34.235353    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8\": container with ID starting with 12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8 not found: ID does not exist" containerID="12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"
Jul 28 01:21:34.235422 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.235372    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8"} err="failed to get container status \"12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8\": rpc error: code = NotFound desc = could not find container \"12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8\": container with ID starting with 12e7b14bd9f5a3dc3098201283406f5dd63aa7a8ba59d01d61ca80990c3d79f8 not found: ID does not exist"
Jul 28 01:21:34.334279 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334232    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-data\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334279 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z7rg\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-kube-api-access-5z7rg\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334299    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-credential-socket\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334340    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cacfa-3690-473c-b8b3-d676f461d103-istio-podinfo\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-certs\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334385    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-envoy\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334433    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-istio-token\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334795 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334532    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-socket\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334795 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334592    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cacfa-3690-473c-b8b3-d676f461d103-istiod-ca-cert\") pod \"344cacfa-3690-473c-b8b3-d676f461d103\" (UID: \"344cacfa-3690-473c-b8b3-d676f461d103\") "
Jul 28 01:21:34.334795 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:21:34.335009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334870    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:21:34.335009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.334980    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.335009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.335002    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.335179 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.335088    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344cacfa-3690-473c-b8b3-d676f461d103-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:21:34.335263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.335242    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:21:34.335325 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.335263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-data" (OuterVolumeSpecName: "istio-data") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:21:34.336702 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.336677    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/344cacfa-3690-473c-b8b3-d676f461d103-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:21:34.336702 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.336694    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-kube-api-access-5z7rg" (OuterVolumeSpecName: "kube-api-access-5z7rg") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "kube-api-access-5z7rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:21:34.336848 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.336703    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-istio-token" (OuterVolumeSpecName: "istio-token") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:21:34.336962 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.336923    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "344cacfa-3690-473c-b8b3-d676f461d103" (UID: "344cacfa-3690-473c-b8b3-d676f461d103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:21:34.435556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435518    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.435556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435548    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cacfa-3690-473c-b8b3-d676f461d103-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.435556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435559    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.435819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435566    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7rg\" (UniqueName: \"kubernetes.io/projected/344cacfa-3690-473c-b8b3-d676f461d103-kube-api-access-5z7rg\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.435819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435577    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.435819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435585    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cacfa-3690-473c-b8b3-d676f461d103-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.435819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.435594    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cacfa-3690-473c-b8b3-d676f461d103-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:21:34.548436 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.548401    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"]
Jul 28 01:21:34.560791 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.560762    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7ce60076-openshift-default-567cb8dc48-spsz4"]
Jul 28 01:21:34.945827 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:34.945798    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="344cacfa-3690-473c-b8b3-d676f461d103" path="/var/lib/kubelet/pods/344cacfa-3690-473c-b8b3-d676f461d103/volumes"
Jul 28 01:21:44.918118 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:21:44.918079    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:22:09.149444 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:09.149368    2564 scope.go:117] "RemoveContainer" containerID="6e90ca8f4f69c21e5fdc21b7b93c799fda9f746a752837f818745dbe967534ae"
Jul 28 01:22:24.592150 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592115    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"]
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592471    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8820f73c-64b5-48a3-aa83-e178a78f1564" containerName="maas-api"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592482    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8820f73c-64b5-48a3-aa83-e178a78f1564" containerName="maas-api"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592489    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c55d25-872c-4b3a-bebe-23c9ade1e373" containerName="revoke-keys"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592495    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c55d25-872c-4b3a-bebe-23c9ade1e373" containerName="revoke-keys"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592507    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592513    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592523    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy"
Jul 28 01:22:24.592534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592528    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy"
Jul 28 01:22:24.592793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592589    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy"
Jul 28 01:22:24.592793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592596    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c55d25-872c-4b3a-bebe-23c9ade1e373" containerName="revoke-keys"
Jul 28 01:22:24.592793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.592603    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8820f73c-64b5-48a3-aa83-e178a78f1564" containerName="maas-api"
Jul 28 01:22:24.595487 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.595468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.597888 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.597866    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ee92ea-openshift-default-dockercfg-dk9ct\""
Jul 28 01:22:24.608826 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.608796    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"]
Jul 28 01:22:24.630420 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-credential-socket\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-podinfo\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-certs\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-data\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630565    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-token\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630842 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp9d\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-kube-api-access-bsp9d\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630842 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-envoy\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630842 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istiod-ca-cert\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.630842 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.630770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-socket\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istiod-ca-cert\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-socket\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-credential-socket\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731328    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-podinfo\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-certs\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-data\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-token\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp9d\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-kube-api-access-bsp9d\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731735 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-envoy\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731791 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-credential-socket\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.731850 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-socket\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.732012 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.731987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-data\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.732098 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.732030    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istiod-ca-cert\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.732214 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.732198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-certs\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.733697 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.733679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-envoy\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.733969 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.733930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-podinfo\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.743831 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.743809    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-token\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.744993 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.744973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp9d\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-kube-api-access-bsp9d\") pod \"e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") " pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:24.906911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:24.906810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:25.054589 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.054561    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"]
Jul 28 01:22:25.056203 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:22:25.056177    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6a8f65_7db7_45bc_9f57_8d78fa003bb9.slice/crio-a2a140d827be248b6c1060ba503e842cb0766aba425dbc0b2775057f2173fe7f WatchSource:0}: Error finding container a2a140d827be248b6c1060ba503e842cb0766aba425dbc0b2775057f2173fe7f: Status 404 returned error can't find the container with id a2a140d827be248b6c1060ba503e842cb0766aba425dbc0b2775057f2173fe7f
Jul 28 01:22:25.058323 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.058288    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:22:25.058404 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.058355    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:22:25.058443 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.058401    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:22:25.470160 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.470121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" event={"ID":"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9","Type":"ContainerStarted","Data":"3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"}
Jul 28 01:22:25.470366 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.470179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" event={"ID":"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9","Type":"ContainerStarted","Data":"a2a140d827be248b6c1060ba503e842cb0766aba425dbc0b2775057f2173fe7f"}
Jul 28 01:22:25.492642 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.492590    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" podStartSLOduration=1.492573186 podStartE2EDuration="1.492573186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:25.490095593 +0000 UTC m=+2177.045598766" watchObservedRunningTime="2026-07-28 01:22:25.492573186 +0000 UTC m=+2177.048076345"
Jul 28 01:22:25.907014 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:25.906917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:26.913278 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:26.913248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:27.479654 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.479622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:27.480860 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.480838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:27.534751 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.534716    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"]
Jul 28 01:22:27.535638 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.535611    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="344cacfa-3690-473c-b8b3-d676f461d103" containerName="istio-proxy"
Jul 28 01:22:27.540022 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.539435    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.542416 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.542390    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ee92ea-serving-cert\""
Jul 28 01:22:27.542846 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.542824    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:22:27.543162 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.543140    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:22:27.543276 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.543228    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:22:27.550793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.549841    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"]
Jul 28 01:22:27.557335 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.557306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.557541 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.557519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd654\" (UniqueName: \"kubernetes.io/projected/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-kube-api-access-kd654\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.557716 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.557688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.659230 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.659198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.659408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.659262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd654\" (UniqueName: \"kubernetes.io/projected/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-kube-api-access-kd654\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.659408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.659302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.659549 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:22:27.659415    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-serving-cert: secret "maas-api-e2e-inf-a-ee92ea-serving-cert" not found
Jul 28 01:22:27.659549 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:22:27.659496    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls podName:fb722d3f-6953-4d10-b2bb-fb75a8eeac46 nodeName:}" failed. No retries permitted until 2026-07-28 01:22:28.159474769 +0000 UTC m=+2179.714977910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls") pod "maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" (UID: "fb722d3f-6953-4d10-b2bb-fb75a8eeac46") : secret "maas-api-e2e-inf-a-ee92ea-serving-cert" not found
Jul 28 01:22:27.660040 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.660009    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:27.668651 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:27.668627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd654\" (UniqueName: \"kubernetes.io/projected/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-kube-api-access-kd654\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:28.164304 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:28.164261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:28.166810 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:28.166783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:28.455556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:28.455518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:28.616204 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:28.616166    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"]
Jul 28 01:22:28.618784 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:22:28.618753    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb722d3f_6953_4d10_b2bb_fb75a8eeac46.slice/crio-ccd70df9b64487a3f62835e501ff6be2ecd16cd0bd2101191d3d79a5fff9e138 WatchSource:0}: Error finding container ccd70df9b64487a3f62835e501ff6be2ecd16cd0bd2101191d3d79a5fff9e138: Status 404 returned error can't find the container with id ccd70df9b64487a3f62835e501ff6be2ecd16cd0bd2101191d3d79a5fff9e138
Jul 28 01:22:29.490514 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:29.490477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" event={"ID":"fb722d3f-6953-4d10-b2bb-fb75a8eeac46","Type":"ContainerStarted","Data":"2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0"}
Jul 28 01:22:29.490929 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:29.490530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" event={"ID":"fb722d3f-6953-4d10-b2bb-fb75a8eeac46","Type":"ContainerStarted","Data":"ccd70df9b64487a3f62835e501ff6be2ecd16cd0bd2101191d3d79a5fff9e138"}
Jul 28 01:22:29.508854 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:29.508803    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" podStartSLOduration=2.081859578 podStartE2EDuration="2.508787309s" totalImagesPullingTime="426.927731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:29.507188121 +0000 UTC m=+2181.062691331" watchObservedRunningTime="2026-07-28 01:22:29.508787309 +0000 UTC m=+2181.064290466"
Jul 28 01:22:30.495864 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:30.495826    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:32.903523 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:32.903483    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:22:35.932613 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:35.932573    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"]
Jul 28 01:22:35.937077 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:35.937057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:35.939556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:35.939528    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f1343b-serving-cert\""
Jul 28 01:22:35.946525 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:35.946501    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"]
Jul 28 01:22:36.043767 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.043727    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.043767 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.043769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6s4z\" (UniqueName: \"kubernetes.io/projected/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-kube-api-access-r6s4z\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.044065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.043870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.144814 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.144767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.145016 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.144817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6s4z\" (UniqueName: \"kubernetes.io/projected/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-kube-api-access-r6s4z\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.145016 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.144861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.145016 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:22:36.144917    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-serving-cert: secret "maas-api-e2e-inf-b-f1343b-serving-cert" not found
Jul 28 01:22:36.145016 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:22:36.145016    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls podName:60ae6d58-d6b0-41e7-be7e-1a03d7e1505a nodeName:}" failed. No retries permitted until 2026-07-28 01:22:36.644993216 +0000 UTC m=+2188.200496373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls") pod "maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" (UID: "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a") : secret "maas-api-e2e-inf-b-f1343b-serving-cert" not found
Jul 28 01:22:36.145546 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.145526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.153649 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.153624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6s4z\" (UniqueName: \"kubernetes.io/projected/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-kube-api-access-r6s4z\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.506079 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.506049    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:22:36.648836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.648804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.651269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.651242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:36.848902 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:36.848821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:37.033317 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:37.033293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"]
Jul 28 01:22:37.035763 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:22:37.035732    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ae6d58_d6b0_41e7_be7e_1a03d7e1505a.slice/crio-954404e16104501a3460d7f1fab8488e88a367a5b96d1d8c1a26fe3ac4d9443b WatchSource:0}: Error finding container 954404e16104501a3460d7f1fab8488e88a367a5b96d1d8c1a26fe3ac4d9443b: Status 404 returned error can't find the container with id 954404e16104501a3460d7f1fab8488e88a367a5b96d1d8c1a26fe3ac4d9443b
Jul 28 01:22:37.529906 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:37.529871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" event={"ID":"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a","Type":"ContainerStarted","Data":"a3c8e3d81820c409efd93e1e49f471299d19f787c0d144b46f324126543e2b01"}
Jul 28 01:22:37.530111 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:37.529922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" event={"ID":"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a","Type":"ContainerStarted","Data":"954404e16104501a3460d7f1fab8488e88a367a5b96d1d8c1a26fe3ac4d9443b"}
Jul 28 01:22:37.530111 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:37.529964    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:41.552119 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.552084    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerID="3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e" exitCode=0
Jul 28 01:22:41.552666 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.552160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" event={"ID":"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9","Type":"ContainerDied","Data":"3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"}
Jul 28 01:22:41.552666 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.552651    2564 scope.go:117] "RemoveContainer" containerID="3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"
Jul 28 01:22:41.553679 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.553644    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:22:41.553802 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.553751    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:22:41.553869 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.553813    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:22:41.574334 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.574280    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" podStartSLOduration=6.22362945 podStartE2EDuration="6.574260288s" totalImagesPullingTime="350.630838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:37.57653754 +0000 UTC m=+2189.132040699" watchObservedRunningTime="2026-07-28 01:22:41.574260288 +0000 UTC m=+2193.129763449"
Jul 28 01:22:41.612477 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:41.612438    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:22:42.480540 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:42.480466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:42.559894 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:42.559855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" event={"ID":"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9","Type":"ContainerStarted","Data":"ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35"}
Jul 28 01:22:42.907606 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:42.907514    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:42.909019 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:42.908990    2564 patch_prober.go:28] interesting pod/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 01:22:42.909156 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:42.909044    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 01:22:43.541311 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:43.541281    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:22:43.907339 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:43.907249    2564 patch_prober.go:28] interesting pod/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 01:22:43.907339 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:43.907308    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 01:22:44.907833 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:44.907796    2564 patch_prober.go:28] interesting pod/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 01:22:44.908250 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:44.907854    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 01:22:45.907425 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:45.907386    2564 patch_prober.go:28] interesting pod/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 01:22:45.907625 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:45.907453    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 01:22:46.912013 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:46.911978    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:46.912505 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:46.912359    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:22:46.913088 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:22:46.913067    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:23:04.405405 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:23:04.405366    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:23:09.175195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:23:09.175165    2564 scope.go:117] "RemoveContainer" containerID="52c7a9f5f929437fe5f7b65808f0ad5d4c86ff6d31ab4b37f71b68f95c3fa1a0"
Jul 28 01:23:54.110533 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:23:54.110488    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:24:07.832590 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:07.832551    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:24:41.709139 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.709094    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"]
Jul 28 01:24:41.711770 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.711744    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:41.713905 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.713882    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:24:41.719788 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.719750    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"]
Jul 28 01:24:41.870390 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.870346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b212acd-ef12-4344-a9b4-a94bdc80b211-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn\" (UID: \"2b212acd-ef12-4344-a9b4-a94bdc80b211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:41.971919 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.971809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b212acd-ef12-4344-a9b4-a94bdc80b211-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn\" (UID: \"2b212acd-ef12-4344-a9b4-a94bdc80b211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:41.972502 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:41.972479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b212acd-ef12-4344-a9b4-a94bdc80b211-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn\" (UID: \"2b212acd-ef12-4344-a9b4-a94bdc80b211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:42.024297 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:42.024235    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:42.161857 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:42.161823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"]
Jul 28 01:24:42.163682 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:24:42.163654    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b212acd_ef12_4344_a9b4_a94bdc80b211.slice/crio-bda87d0f8df64dd9b883a3cdbb9b02abf6d0b1b379d729e7d707a0b76ee4a7b6 WatchSource:0}: Error finding container bda87d0f8df64dd9b883a3cdbb9b02abf6d0b1b379d729e7d707a0b76ee4a7b6: Status 404 returned error can't find the container with id bda87d0f8df64dd9b883a3cdbb9b02abf6d0b1b379d729e7d707a0b76ee4a7b6
Jul 28 01:24:42.186609 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:42.186579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn" event={"ID":"2b212acd-ef12-4344-a9b4-a94bdc80b211","Type":"ContainerStarted","Data":"bda87d0f8df64dd9b883a3cdbb9b02abf6d0b1b379d729e7d707a0b76ee4a7b6"}
Jul 28 01:24:43.193057 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:43.193022    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b212acd-ef12-4344-a9b4-a94bdc80b211" containerID="db89f84fc621682f68de4f4e598351284c03d06c68894db7d2121ebad6739d46" exitCode=0
Jul 28 01:24:43.193476 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:43.193111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn" event={"ID":"2b212acd-ef12-4344-a9b4-a94bdc80b211","Type":"ContainerDied","Data":"db89f84fc621682f68de4f4e598351284c03d06c68894db7d2121ebad6739d46"}
Jul 28 01:24:44.335227 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:44.335200    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:44.492927 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:44.492831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b212acd-ef12-4344-a9b4-a94bdc80b211-maas-api-service-ca\") pod \"2b212acd-ef12-4344-a9b4-a94bdc80b211\" (UID: \"2b212acd-ef12-4344-a9b4-a94bdc80b211\") "
Jul 28 01:24:44.493232 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:44.493205    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b212acd-ef12-4344-a9b4-a94bdc80b211-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b212acd-ef12-4344-a9b4-a94bdc80b211" (UID: "2b212acd-ef12-4344-a9b4-a94bdc80b211"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:24:44.594266 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:44.594224    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b212acd-ef12-4344-a9b4-a94bdc80b211-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:24:45.204492 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:45.204403    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"
Jul 28 01:24:45.204492 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:45.204451    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn" event={"ID":"2b212acd-ef12-4344-a9b4-a94bdc80b211","Type":"ContainerDied","Data":"bda87d0f8df64dd9b883a3cdbb9b02abf6d0b1b379d729e7d707a0b76ee4a7b6"}
Jul 28 01:24:45.204732 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:45.204504    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda87d0f8df64dd9b883a3cdbb9b02abf6d0b1b379d729e7d707a0b76ee4a7b6"
Jul 28 01:24:45.700517 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:45.700476    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:24:52.205735 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.205702    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"]
Jul 28 01:24:52.206201 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.205909    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" podUID="fb722d3f-6953-4d10-b2bb-fb75a8eeac46" containerName="maas-api" containerID="cri-o://2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0" gracePeriod=30
Jul 28 01:24:52.463356 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.463293    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:24:52.564091 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.564052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls\") pod \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") "
Jul 28 01:24:52.564264 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.564134    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd654\" (UniqueName: \"kubernetes.io/projected/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-kube-api-access-kd654\") pod \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") "
Jul 28 01:24:52.564264 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.564177    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-openshift-service-ca\") pod \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\" (UID: \"fb722d3f-6953-4d10-b2bb-fb75a8eeac46\") "
Jul 28 01:24:52.564605 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.564573    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb722d3f-6953-4d10-b2bb-fb75a8eeac46" (UID: "fb722d3f-6953-4d10-b2bb-fb75a8eeac46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:24:52.566273 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.566239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb722d3f-6953-4d10-b2bb-fb75a8eeac46" (UID: "fb722d3f-6953-4d10-b2bb-fb75a8eeac46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:24:52.566365 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.566268    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-kube-api-access-kd654" (OuterVolumeSpecName: "kube-api-access-kd654") pod "fb722d3f-6953-4d10-b2bb-fb75a8eeac46" (UID: "fb722d3f-6953-4d10-b2bb-fb75a8eeac46"). InnerVolumeSpecName "kube-api-access-kd654". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:24:52.664893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.664852    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:24:52.664893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.664888    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd654\" (UniqueName: \"kubernetes.io/projected/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-kube-api-access-kd654\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:24:52.664893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:52.664902    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb722d3f-6953-4d10-b2bb-fb75a8eeac46-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:24:53.244312 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.244221    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb722d3f-6953-4d10-b2bb-fb75a8eeac46" containerID="2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0" exitCode=0
Jul 28 01:24:53.244312 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.244289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" event={"ID":"fb722d3f-6953-4d10-b2bb-fb75a8eeac46","Type":"ContainerDied","Data":"2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0"}
Jul 28 01:24:53.244312 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.244297    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"
Jul 28 01:24:53.244880 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.244330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8" event={"ID":"fb722d3f-6953-4d10-b2bb-fb75a8eeac46","Type":"ContainerDied","Data":"ccd70df9b64487a3f62835e501ff6be2ecd16cd0bd2101191d3d79a5fff9e138"}
Jul 28 01:24:53.244880 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.244352    2564 scope.go:117] "RemoveContainer" containerID="2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0"
Jul 28 01:24:53.253338 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.253316    2564 scope.go:117] "RemoveContainer" containerID="2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0"
Jul 28 01:24:53.253595 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:24:53.253575    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0\": container with ID starting with 2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0 not found: ID does not exist" containerID="2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0"
Jul 28 01:24:53.253674 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.253608    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0"} err="failed to get container status \"2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0\": rpc error: code = NotFound desc = could not find container \"2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0\": container with ID starting with 2755096f29a0cf58b05b7d733ca2a47547c5ec0b3dd95017cf693af3022bcee0 not found: ID does not exist"
Jul 28 01:24:53.262072 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.262043    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"]
Jul 28 01:24:53.264906 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:53.264881    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee92ea-bc595546-9sfj8"]
Jul 28 01:24:54.944925 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:54.944892    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb722d3f-6953-4d10-b2bb-fb75a8eeac46" path="/var/lib/kubelet/pods/fb722d3f-6953-4d10-b2bb-fb75a8eeac46/volumes"
Jul 28 01:24:56.812278 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:56.812236    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"]
Jul 28 01:24:56.814799 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:56.814770    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ee92ea-882jn"]
Jul 28 01:24:56.945128 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:56.945096    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b212acd-ef12-4344-a9b4-a94bdc80b211" path="/var/lib/kubelet/pods/2b212acd-ef12-4344-a9b4-a94bdc80b211/volumes"
Jul 28 01:24:57.005651 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:24:57.005618    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:25:03.335075 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.334976    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"]
Jul 28 01:25:03.335519 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.335241    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy" containerID="cri-o://ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35" gracePeriod=30
Jul 28 01:25:03.685661 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.685618    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"]
Jul 28 01:25:03.686108 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.686089    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b212acd-ef12-4344-a9b4-a94bdc80b211" containerName="revoke-keys"
Jul 28 01:25:03.686217 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.686110    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b212acd-ef12-4344-a9b4-a94bdc80b211" containerName="revoke-keys"
Jul 28 01:25:03.686217 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.686145    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb722d3f-6953-4d10-b2bb-fb75a8eeac46" containerName="maas-api"
Jul 28 01:25:03.686217 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.686156    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb722d3f-6953-4d10-b2bb-fb75a8eeac46" containerName="maas-api"
Jul 28 01:25:03.686363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.686241    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b212acd-ef12-4344-a9b4-a94bdc80b211" containerName="revoke-keys"
Jul 28 01:25:03.686363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.686256    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb722d3f-6953-4d10-b2bb-fb75a8eeac46" containerName="maas-api"
Jul 28 01:25:03.689033 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.689014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:03.691267 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.691247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:25:03.697081 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.695978    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"]
Jul 28 01:25:03.753189 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.753154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b09d6c-bc57-4dac-87e4-05291762d9f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9\" (UID: \"81b09d6c-bc57-4dac-87e4-05291762d9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:03.854137 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.854092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b09d6c-bc57-4dac-87e4-05291762d9f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9\" (UID: \"81b09d6c-bc57-4dac-87e4-05291762d9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:03.854760 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:03.854736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b09d6c-bc57-4dac-87e4-05291762d9f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9\" (UID: \"81b09d6c-bc57-4dac-87e4-05291762d9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:04.001102 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:04.001011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:04.138638 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:04.138610    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"]
Jul 28 01:25:04.141314 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:25:04.141283    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b09d6c_bc57_4dac_87e4_05291762d9f0.slice/crio-5c035a4138dbc34011add8b454d05daa5510ccd352415dfada945deb4c0fe949 WatchSource:0}: Error finding container 5c035a4138dbc34011add8b454d05daa5510ccd352415dfada945deb4c0fe949: Status 404 returned error can't find the container with id 5c035a4138dbc34011add8b454d05daa5510ccd352415dfada945deb4c0fe949
Jul 28 01:25:04.301000 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:04.300887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9" event={"ID":"81b09d6c-bc57-4dac-87e4-05291762d9f0","Type":"ContainerStarted","Data":"5c035a4138dbc34011add8b454d05daa5510ccd352415dfada945deb4c0fe949"}
Jul 28 01:25:05.309502 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:05.309461    2564 generic.go:356] "Generic (PLEG): container finished" podID="81b09d6c-bc57-4dac-87e4-05291762d9f0" containerID="56484b09fe3926eb79647e54d2ff6c596e9879428e50089180a8fcea5eeae16a" exitCode=0
Jul 28 01:25:05.309965 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:05.309551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9" event={"ID":"81b09d6c-bc57-4dac-87e4-05291762d9f0","Type":"ContainerDied","Data":"56484b09fe3926eb79647e54d2ff6c596e9879428e50089180a8fcea5eeae16a"}
Jul 28 01:25:06.458893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:06.458863    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:06.480356 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:06.480320    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b09d6c-bc57-4dac-87e4-05291762d9f0-maas-api-service-ca\") pod \"81b09d6c-bc57-4dac-87e4-05291762d9f0\" (UID: \"81b09d6c-bc57-4dac-87e4-05291762d9f0\") "
Jul 28 01:25:06.480694 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:06.480668    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b09d6c-bc57-4dac-87e4-05291762d9f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81b09d6c-bc57-4dac-87e4-05291762d9f0" (UID: "81b09d6c-bc57-4dac-87e4-05291762d9f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:25:06.581380 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:06.581282    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b09d6c-bc57-4dac-87e4-05291762d9f0-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:07.322496 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:07.322416    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"
Jul 28 01:25:07.322496 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:07.322432    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9" event={"ID":"81b09d6c-bc57-4dac-87e4-05291762d9f0","Type":"ContainerDied","Data":"5c035a4138dbc34011add8b454d05daa5510ccd352415dfada945deb4c0fe949"}
Jul 28 01:25:07.322496 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:07.322480    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c035a4138dbc34011add8b454d05daa5510ccd352415dfada945deb4c0fe949"
Jul 28 01:25:08.597571 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.597540    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:25:08.701683 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.701648    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsp9d\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-kube-api-access-bsp9d\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.701857 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.701704    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-certs\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.701857 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.701746    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-data\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.701857 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.701810    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-podinfo\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.701857 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.701835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-envoy\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.702075 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.701876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-socket\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.702075 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702032    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-token\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.702180 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702076    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-credential-socket\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.702180 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702079    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:25:08.702180 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istiod-ca-cert\") pod \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\" (UID: \"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9\") "
Jul 28 01:25:08.702334 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702168    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:25:08.702611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702414    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.702611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702438    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.702611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702557    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:25:08.702611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702571    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:25:08.702823 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.702758    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:25:08.704185 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.704159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:25:08.704282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.704233    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-kube-api-access-bsp9d" (OuterVolumeSpecName: "kube-api-access-bsp9d") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "kube-api-access-bsp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:25:08.704432 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.704412    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:25:08.704484 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.704418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" (UID: "2b6a8f65-7db7-45bc-9f57-8d78fa003bb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:25:08.803254 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803212    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.803254 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803244    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.803254 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803260    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.803529 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803270    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.803529 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803284    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.803529 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803298    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsp9d\" (UniqueName: \"kubernetes.io/projected/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-kube-api-access-bsp9d\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:08.803529 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:08.803309    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:09.201519 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.201482    2564 scope.go:117] "RemoveContainer" containerID="1244ad47ff2cf3f11b2ec4f19a85319d00f18e7d461deba0b5a7274c1ff32d01"
Jul 28 01:25:09.333113 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.333077    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerID="ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35" exitCode=0
Jul 28 01:25:09.333293 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.333132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" event={"ID":"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9","Type":"ContainerDied","Data":"ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35"}
Jul 28 01:25:09.333293 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.333138    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"
Jul 28 01:25:09.333293 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.333174    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv" event={"ID":"2b6a8f65-7db7-45bc-9f57-8d78fa003bb9","Type":"ContainerDied","Data":"a2a140d827be248b6c1060ba503e842cb0766aba425dbc0b2775057f2173fe7f"}
Jul 28 01:25:09.333293 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.333195    2564 scope.go:117] "RemoveContainer" containerID="ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35"
Jul 28 01:25:09.347423 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.347395    2564 scope.go:117] "RemoveContainer" containerID="3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"
Jul 28 01:25:09.353481 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.353448    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"]
Jul 28 01:25:09.356736 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.356708    2564 scope.go:117] "RemoveContainer" containerID="ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35"
Jul 28 01:25:09.357064 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:25:09.357042    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35\": container with ID starting with ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35 not found: ID does not exist" containerID="ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35"
Jul 28 01:25:09.357158 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.357077    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35"} err="failed to get container status \"ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35\": rpc error: code = NotFound desc = could not find container \"ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35\": container with ID starting with ff11052394c14d1b5d9068c079b919cec1050128f062cd9b5eb531d2d0f10a35 not found: ID does not exist"
Jul 28 01:25:09.357158 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.357104    2564 scope.go:117] "RemoveContainer" containerID="3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"
Jul 28 01:25:09.357422 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:25:09.357404    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e\": container with ID starting with 3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e not found: ID does not exist" containerID="3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"
Jul 28 01:25:09.357463 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.357428    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e"} err="failed to get container status \"3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e\": rpc error: code = NotFound desc = could not find container \"3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e\": container with ID starting with 3d80951f1c351967ecfac79bddffbb289a7f51677d6202e2be98f712ed60ad0e not found: ID does not exist"
Jul 28 01:25:09.357613 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:09.357597    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee92ea-openshift-default-5cf9d95f44-r7jbv"]
Jul 28 01:25:10.944999 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:10.944969    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" path="/var/lib/kubelet/pods/2b6a8f65-7db7-45bc-9f57-8d78fa003bb9/volumes"
Jul 28 01:25:14.191037 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.190993    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"]
Jul 28 01:25:14.191702 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.191292    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" podUID="60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" containerName="maas-api" containerID="cri-o://a3c8e3d81820c409efd93e1e49f471299d19f787c0d144b46f324126543e2b01" gracePeriod=30
Jul 28 01:25:14.359923 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.359888    2564 generic.go:356] "Generic (PLEG): container finished" podID="60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" containerID="a3c8e3d81820c409efd93e1e49f471299d19f787c0d144b46f324126543e2b01" exitCode=0
Jul 28 01:25:14.360082 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.359975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" event={"ID":"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a","Type":"ContainerDied","Data":"a3c8e3d81820c409efd93e1e49f471299d19f787c0d144b46f324126543e2b01"}
Jul 28 01:25:14.442670 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.442607    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:25:14.567801 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.567764    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls\") pod \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") "
Jul 28 01:25:14.568021 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.567806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6s4z\" (UniqueName: \"kubernetes.io/projected/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-kube-api-access-r6s4z\") pod \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") "
Jul 28 01:25:14.568021 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.567880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-openshift-service-ca\") pod \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\" (UID: \"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a\") "
Jul 28 01:25:14.568291 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.568262    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" (UID: "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:25:14.570268 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.570237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" (UID: "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:25:14.570404 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.570235    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-kube-api-access-r6s4z" (OuterVolumeSpecName: "kube-api-access-r6s4z") pod "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" (UID: "60ae6d58-d6b0-41e7-be7e-1a03d7e1505a"). InnerVolumeSpecName "kube-api-access-r6s4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:25:14.668893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.668858    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:14.668893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.668889    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6s4z\" (UniqueName: \"kubernetes.io/projected/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-kube-api-access-r6s4z\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:14.668893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:14.668900    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:25:15.367012 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:15.366912    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"
Jul 28 01:25:15.367012 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:15.366967    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf" event={"ID":"60ae6d58-d6b0-41e7-be7e-1a03d7e1505a","Type":"ContainerDied","Data":"954404e16104501a3460d7f1fab8488e88a367a5b96d1d8c1a26fe3ac4d9443b"}
Jul 28 01:25:15.367513 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:15.367042    2564 scope.go:117] "RemoveContainer" containerID="a3c8e3d81820c409efd93e1e49f471299d19f787c0d144b46f324126543e2b01"
Jul 28 01:25:15.382489 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:15.382460    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"]
Jul 28 01:25:15.386194 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:15.386168    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1343b-86869d9dd9-7hrcf"]
Jul 28 01:25:16.945836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:16.945804    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" path="/var/lib/kubelet/pods/60ae6d58-d6b0-41e7-be7e-1a03d7e1505a/volumes"
Jul 28 01:25:17.366292 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:17.366205    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:25:18.766288 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:18.766253    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"]
Jul 28 01:25:18.768985 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:18.768961    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f1343b-zjrn9"]
Jul 28 01:25:18.945438 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:18.945406    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b09d6c-bc57-4dac-87e4-05291762d9f0" path="/var/lib/kubelet/pods/81b09d6c-bc57-4dac-87e4-05291762d9f0/volumes"
Jul 28 01:25:26.082791 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.082749    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"]
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083125    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" containerName="maas-api"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083137    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" containerName="maas-api"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083149    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083155    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083171    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083177    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083188    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b09d6c-bc57-4dac-87e4-05291762d9f0" containerName="revoke-keys"
Jul 28 01:25:26.083195 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083193    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b09d6c-bc57-4dac-87e4-05291762d9f0" containerName="revoke-keys"
Jul 28 01:25:26.083445 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083246    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy"
Jul 28 01:25:26.083445 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083254    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ae6d58-d6b0-41e7-be7e-1a03d7e1505a" containerName="maas-api"
Jul 28 01:25:26.083445 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083262    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6a8f65-7db7-45bc-9f57-8d78fa003bb9" containerName="istio-proxy"
Jul 28 01:25:26.083445 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.083268    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b09d6c-bc57-4dac-87e4-05291762d9f0" containerName="revoke-keys"
Jul 28 01:25:26.087324 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.087303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.089835 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.089812    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5222f2-openshift-default-dockercfg-f6hd5\""
Jul 28 01:25:26.096753 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.096719    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"]
Jul 28 01:25:26.179560 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-data\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.179805 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-certs\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.179805 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9nr\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-kube-api-access-fp9nr\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.179805 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-credential-socket\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.179805 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-envoy\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.180100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-socket\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.180100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552257c-00a4-47bf-b995-d4b07cbafdd6-istiod-ca-cert\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.180100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-token\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.180100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.179910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-podinfo\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.280786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-socket\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.280786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552257c-00a4-47bf-b995-d4b07cbafdd6-istiod-ca-cert\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-token\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-podinfo\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-data\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-certs\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9nr\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-kube-api-access-fp9nr\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.280921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-credential-socket\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.281280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-socket\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.281311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-credential-socket\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.281336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-certs\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.281405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-envoy\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281608 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.281584    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-data\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.281744 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.281720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552257c-00a4-47bf-b995-d4b07cbafdd6-istiod-ca-cert\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.283440 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.283417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-envoy\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.283522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.283446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-podinfo\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.290063 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.290038    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-token\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.290162 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.290093    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9nr\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-kube-api-access-fp9nr\") pod \"e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") " pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.400024 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.399918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:26.544306 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.544278    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"]
Jul 28 01:25:26.546417 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:25:26.546382    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1552257c_00a4_47bf_b995_d4b07cbafdd6.slice/crio-fa3f45c8fa3f04cc56a9d9e572d80eae271f0af15bcac21d9975d54151595a75 WatchSource:0}: Error finding container fa3f45c8fa3f04cc56a9d9e572d80eae271f0af15bcac21d9975d54151595a75: Status 404 returned error can't find the container with id fa3f45c8fa3f04cc56a9d9e572d80eae271f0af15bcac21d9975d54151595a75
Jul 28 01:25:26.548718 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.548679    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:25:26.548799 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.548769    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:25:26.548838 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:26.548818    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:25:27.429165 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:27.429131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" event={"ID":"1552257c-00a4-47bf-b995-d4b07cbafdd6","Type":"ContainerStarted","Data":"d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"}
Jul 28 01:25:27.429594 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:27.429185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" event={"ID":"1552257c-00a4-47bf-b995-d4b07cbafdd6","Type":"ContainerStarted","Data":"fa3f45c8fa3f04cc56a9d9e572d80eae271f0af15bcac21d9975d54151595a75"}
Jul 28 01:25:27.450595 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:27.450536    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podStartSLOduration=1.4505200710000001 podStartE2EDuration="1.450520071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:27.447982867 +0000 UTC m=+2359.003486053" watchObservedRunningTime="2026-07-28 01:25:27.450520071 +0000 UTC m=+2359.006023230"
Jul 28 01:25:28.400247 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:28.400200    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:28.405176 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:28.405151    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:28.435332 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:28.435296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:28.436329 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:28.436308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:29.520781 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.520748    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"]
Jul 28 01:25:29.524241 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.524219    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.527269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.527241    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:25:29.527789 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.527767    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5222f2-serving-cert\""
Jul 28 01:25:29.527906 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.527770    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:25:29.527906 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.527770    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:25:29.540603 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.540577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"]
Jul 28 01:25:29.710286 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.710250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668e67e8-31d6-43ac-947d-ff06193a2196-maas-api-tls\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.710286 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.710287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/668e67e8-31d6-43ac-947d-ff06193a2196-kube-api-access-n9czt\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.710534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.710322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e67e8-31d6-43ac-947d-ff06193a2196-openshift-service-ca\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.811215 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.811116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e67e8-31d6-43ac-947d-ff06193a2196-openshift-service-ca\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.811401 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.811238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668e67e8-31d6-43ac-947d-ff06193a2196-maas-api-tls\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.811401 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.811271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/668e67e8-31d6-43ac-947d-ff06193a2196-kube-api-access-n9czt\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.811780 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.811755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e67e8-31d6-43ac-947d-ff06193a2196-openshift-service-ca\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.813799 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.813770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668e67e8-31d6-43ac-947d-ff06193a2196-maas-api-tls\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.828685 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.828647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/668e67e8-31d6-43ac-947d-ff06193a2196-kube-api-access-n9czt\") pod \"maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.837270 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.837248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:29.980309 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.980285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"]
Jul 28 01:25:29.982214 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:25:29.982187    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668e67e8_31d6_43ac_947d_ff06193a2196.slice/crio-0afaf8c7889d1d5bed63e307229462c951844bd92ebe13b55e0e59247fa0fe8a WatchSource:0}: Error finding container 0afaf8c7889d1d5bed63e307229462c951844bd92ebe13b55e0e59247fa0fe8a: Status 404 returned error can't find the container with id 0afaf8c7889d1d5bed63e307229462c951844bd92ebe13b55e0e59247fa0fe8a
Jul 28 01:25:29.983618 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:29.983599    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:30.445331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:30.445291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589" event={"ID":"668e67e8-31d6-43ac-947d-ff06193a2196","Type":"ContainerStarted","Data":"0afaf8c7889d1d5bed63e307229462c951844bd92ebe13b55e0e59247fa0fe8a"}
Jul 28 01:25:31.451973 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:31.451914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589" event={"ID":"668e67e8-31d6-43ac-947d-ff06193a2196","Type":"ContainerStarted","Data":"9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047"}
Jul 28 01:25:31.452382 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:31.452007    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:31.469795 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:31.469745    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589" podStartSLOduration=1.581870287 podStartE2EDuration="2.469728474s" totalImagesPullingTime="887.858187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:31.468040722 +0000 UTC m=+2363.023543905" watchObservedRunningTime="2026-07-28 01:25:31.469728474 +0000 UTC m=+2363.025231654"
Jul 28 01:25:34.907407 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:34.907370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:25:37.475288 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:37.475241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:25:37.996273 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:37.996228    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"]
Jul 28 01:25:38.000503 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.000480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.002798 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.002774    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9c8614-serving-cert\""
Jul 28 01:25:38.024264 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.024229    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"]
Jul 28 01:25:38.088579 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.088518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ad4cc3-ac79-4ec5-897e-80af94996974-openshift-service-ca\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.088792 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.088639    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ad4cc3-ac79-4ec5-897e-80af94996974-maas-api-tls\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.088792 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.088676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzvc\" (UniqueName: \"kubernetes.io/projected/75ad4cc3-ac79-4ec5-897e-80af94996974-kube-api-access-bfzvc\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.189483 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.189439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ad4cc3-ac79-4ec5-897e-80af94996974-maas-api-tls\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.189739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.189492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzvc\" (UniqueName: \"kubernetes.io/projected/75ad4cc3-ac79-4ec5-897e-80af94996974-kube-api-access-bfzvc\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.189739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.189574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ad4cc3-ac79-4ec5-897e-80af94996974-openshift-service-ca\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.190555 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.190520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ad4cc3-ac79-4ec5-897e-80af94996974-openshift-service-ca\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.192626 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.192596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ad4cc3-ac79-4ec5-897e-80af94996974-maas-api-tls\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.202863 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.202836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzvc\" (UniqueName: \"kubernetes.io/projected/75ad4cc3-ac79-4ec5-897e-80af94996974-kube-api-access-bfzvc\") pod \"maas-api-e2e-api-b-9c8614-587c94497b-9zrrg\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.317990 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.317873    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:38.490638 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:38.490561    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"]
Jul 28 01:25:38.493605 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:25:38.493573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ad4cc3_ac79_4ec5_897e_80af94996974.slice/crio-fd1e7600b576aa69471652cc2fb762fad8b308d0f620d32bd4b0b861496e5187 WatchSource:0}: Error finding container fd1e7600b576aa69471652cc2fb762fad8b308d0f620d32bd4b0b861496e5187: Status 404 returned error can't find the container with id fd1e7600b576aa69471652cc2fb762fad8b308d0f620d32bd4b0b861496e5187
Jul 28 01:25:39.504926 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:39.504883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg" event={"ID":"75ad4cc3-ac79-4ec5-897e-80af94996974","Type":"ContainerStarted","Data":"51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302"}
Jul 28 01:25:39.505389 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:39.504968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg" event={"ID":"75ad4cc3-ac79-4ec5-897e-80af94996974","Type":"ContainerStarted","Data":"fd1e7600b576aa69471652cc2fb762fad8b308d0f620d32bd4b0b861496e5187"}
Jul 28 01:25:39.505389 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:39.504996    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:39.521862 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:39.521802    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg" podStartSLOduration=2.015332155 podStartE2EDuration="2.521787903s" totalImagesPullingTime="506.455748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:39.519695525 +0000 UTC m=+2371.075198715" watchObservedRunningTime="2026-07-28 01:25:39.521787903 +0000 UTC m=+2371.077291095"
Jul 28 01:25:43.437600 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.437561    2564 patch_prober.go:28] interesting pod/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 28 01:25:43.438058 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.437633    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 28 01:25:43.528616 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.528527    2564 generic.go:356] "Generic (PLEG): container finished" podID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerID="d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3" exitCode=0
Jul 28 01:25:43.528771 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.528606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" event={"ID":"1552257c-00a4-47bf-b995-d4b07cbafdd6","Type":"ContainerDied","Data":"d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"}
Jul 28 01:25:43.529067 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.529052    2564 scope.go:117] "RemoveContainer" containerID="d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"
Jul 28 01:25:43.535152 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.535102    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:25:43.535480 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.535441    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:25:43.535763 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:43.535728    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:25:44.535000 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:44.534964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" event={"ID":"1552257c-00a4-47bf-b995-d4b07cbafdd6","Type":"ContainerStarted","Data":"1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e"}
Jul 28 01:25:45.400302 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:45.400262    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:45.400558 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:45.400478    2564 patch_prober.go:28] interesting pod/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 28 01:25:45.400558 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:45.400529    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 28 01:25:45.516326 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:45.516296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:25:45.713522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:45.713483    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:25:46.400951 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:46.400896    2564 patch_prober.go:28] interesting pod/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 28 01:25:46.401132 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:46.400964    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 28 01:25:47.401139 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:47.401103    2564 patch_prober.go:28] interesting pod/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 28 01:25:47.401559 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:47.401164    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 28 01:25:48.400632 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:48.400596    2564 patch_prober.go:28] interesting pod/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 28 01:25:48.400830 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:48.400654    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 28 01:25:49.404432 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:49.404398    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:49.404904 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:49.404696    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:25:49.405464 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:25:49.405447    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:26:06.284607 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.284566    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"]
Jul 28 01:26:06.285152 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.284824    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589" podUID="668e67e8-31d6-43ac-947d-ff06193a2196" containerName="maas-api" containerID="cri-o://9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047" gracePeriod=30
Jul 28 01:26:06.557811 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.557780    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:26:06.642771 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.641989    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668e67e8-31d6-43ac-947d-ff06193a2196-maas-api-tls\") pod \"668e67e8-31d6-43ac-947d-ff06193a2196\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") "
Jul 28 01:26:06.642771 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.642067    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/668e67e8-31d6-43ac-947d-ff06193a2196-kube-api-access-n9czt\") pod \"668e67e8-31d6-43ac-947d-ff06193a2196\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") "
Jul 28 01:26:06.642771 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.642133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e67e8-31d6-43ac-947d-ff06193a2196-openshift-service-ca\") pod \"668e67e8-31d6-43ac-947d-ff06193a2196\" (UID: \"668e67e8-31d6-43ac-947d-ff06193a2196\") "
Jul 28 01:26:06.643602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.643280    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668e67e8-31d6-43ac-947d-ff06193a2196-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "668e67e8-31d6-43ac-947d-ff06193a2196" (UID: "668e67e8-31d6-43ac-947d-ff06193a2196"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:06.647804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.646237    2564 generic.go:356] "Generic (PLEG): container finished" podID="668e67e8-31d6-43ac-947d-ff06193a2196" containerID="9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047" exitCode=0
Jul 28 01:26:06.647804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.646358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589" event={"ID":"668e67e8-31d6-43ac-947d-ff06193a2196","Type":"ContainerDied","Data":"9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047"}
Jul 28 01:26:06.647804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.646400    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668e67e8-31d6-43ac-947d-ff06193a2196-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "668e67e8-31d6-43ac-947d-ff06193a2196" (UID: "668e67e8-31d6-43ac-947d-ff06193a2196"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:26:06.647804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.646453    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589" event={"ID":"668e67e8-31d6-43ac-947d-ff06193a2196","Type":"ContainerDied","Data":"0afaf8c7889d1d5bed63e307229462c951844bd92ebe13b55e0e59247fa0fe8a"}
Jul 28 01:26:06.647804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.646496    2564 scope.go:117] "RemoveContainer" containerID="9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047"
Jul 28 01:26:06.647804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.646531    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"
Jul 28 01:26:06.650548 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.650512    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668e67e8-31d6-43ac-947d-ff06193a2196-kube-api-access-n9czt" (OuterVolumeSpecName: "kube-api-access-n9czt") pod "668e67e8-31d6-43ac-947d-ff06193a2196" (UID: "668e67e8-31d6-43ac-947d-ff06193a2196"). InnerVolumeSpecName "kube-api-access-n9czt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:06.667455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.667380    2564 scope.go:117] "RemoveContainer" containerID="9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047"
Jul 28 01:26:06.667851 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:26:06.667824    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047\": container with ID starting with 9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047 not found: ID does not exist" containerID="9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047"
Jul 28 01:26:06.667982 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.667862    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047"} err="failed to get container status \"9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047\": rpc error: code = NotFound desc = could not find container \"9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047\": container with ID starting with 9756c6a871893622b4621151fda1b3f5aea5cae888d41e85ac2a2743a38d5047 not found: ID does not exist"
Jul 28 01:26:06.743739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.743698    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e67e8-31d6-43ac-947d-ff06193a2196-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:06.743739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.743732    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668e67e8-31d6-43ac-947d-ff06193a2196-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:06.743739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.743742    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/668e67e8-31d6-43ac-947d-ff06193a2196-kube-api-access-n9czt\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:06.972311 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.972271    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"]
Jul 28 01:26:06.976253 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:06.976222    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5222f2-7c897fc4bc-gg589"]
Jul 28 01:26:08.808925 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:08.808889    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:26:08.945548 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:08.945513    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668e67e8-31d6-43ac-947d-ff06193a2196" path="/var/lib/kubelet/pods/668e67e8-31d6-43ac-947d-ff06193a2196/volumes"
Jul 28 01:26:09.058504 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:09.058475    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:26:09.067251 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:09.067161    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:26:17.234822 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.234784    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"]
Jul 28 01:26:17.235370 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.235018    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" containerID="cri-o://1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e" gracePeriod=30
Jul 28 01:26:17.580447 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.580348    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"]
Jul 28 01:26:17.581176 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.581150    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668e67e8-31d6-43ac-947d-ff06193a2196" containerName="maas-api"
Jul 28 01:26:17.581176 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.581178    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="668e67e8-31d6-43ac-947d-ff06193a2196" containerName="maas-api"
Jul 28 01:26:17.581389 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.581375    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="668e67e8-31d6-43ac-947d-ff06193a2196" containerName="maas-api"
Jul 28 01:26:17.585248 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.585229    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:17.587845 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.587817    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:26:17.588483 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.588453    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"]
Jul 28 01:26:17.745737 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.745696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745483bb-d5c8-451c-ad87-fa62407985c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz\" (UID: \"745483bb-d5c8-451c-ad87-fa62407985c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:17.846925 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.846829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745483bb-d5c8-451c-ad87-fa62407985c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz\" (UID: \"745483bb-d5c8-451c-ad87-fa62407985c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:17.847505 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.847484    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745483bb-d5c8-451c-ad87-fa62407985c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz\" (UID: \"745483bb-d5c8-451c-ad87-fa62407985c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:17.897906 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:17.897862    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:18.037327 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:18.037201    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"]
Jul 28 01:26:18.039703 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:26:18.039672    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745483bb_d5c8_451c_ad87_fa62407985c2.slice/crio-885964179e9ede61214a186200dd1641658e5ef46974aadab36c8896bf64c4e1 WatchSource:0}: Error finding container 885964179e9ede61214a186200dd1641658e5ef46974aadab36c8896bf64c4e1: Status 404 returned error can't find the container with id 885964179e9ede61214a186200dd1641658e5ef46974aadab36c8896bf64c4e1
Jul 28 01:26:18.708807 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:18.708771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz" event={"ID":"745483bb-d5c8-451c-ad87-fa62407985c2","Type":"ContainerStarted","Data":"b10991e2aadc2c030eaf2cbd07920fd14c35942b40531aac08a91f4170a81a9c"}
Jul 28 01:26:18.709286 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:18.708828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz" event={"ID":"745483bb-d5c8-451c-ad87-fa62407985c2","Type":"ContainerStarted","Data":"885964179e9ede61214a186200dd1641658e5ef46974aadab36c8896bf64c4e1"}
Jul 28 01:26:18.725264 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:18.725213    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz" podStartSLOduration=1.725195998 podStartE2EDuration="1.725195998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:18.724326362 +0000 UTC m=+2410.279829547" watchObservedRunningTime="2026-07-28 01:26:18.725195998 +0000 UTC m=+2410.280699158"
Jul 28 01:26:19.407480 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:19.407437    2564 patch_prober.go:28] interesting pod/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 01:26:19.407646 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:19.407500    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:26:19.714884 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:19.714848    2564 generic.go:356] "Generic (PLEG): container finished" podID="745483bb-d5c8-451c-ad87-fa62407985c2" containerID="b10991e2aadc2c030eaf2cbd07920fd14c35942b40531aac08a91f4170a81a9c" exitCode=0
Jul 28 01:26:19.715322 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:19.714920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz" event={"ID":"745483bb-d5c8-451c-ad87-fa62407985c2","Type":"ContainerDied","Data":"b10991e2aadc2c030eaf2cbd07920fd14c35942b40531aac08a91f4170a81a9c"}
Jul 28 01:26:20.852517 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:20.852489    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:20.977327 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:20.977242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745483bb-d5c8-451c-ad87-fa62407985c2-maas-api-service-ca\") pod \"745483bb-d5c8-451c-ad87-fa62407985c2\" (UID: \"745483bb-d5c8-451c-ad87-fa62407985c2\") "
Jul 28 01:26:20.977640 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:20.977614    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745483bb-d5c8-451c-ad87-fa62407985c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "745483bb-d5c8-451c-ad87-fa62407985c2" (UID: "745483bb-d5c8-451c-ad87-fa62407985c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:21.078922 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:21.078888    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745483bb-d5c8-451c-ad87-fa62407985c2-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:21.726068 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:21.726030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz" event={"ID":"745483bb-d5c8-451c-ad87-fa62407985c2","Type":"ContainerDied","Data":"885964179e9ede61214a186200dd1641658e5ef46974aadab36c8896bf64c4e1"}
Jul 28 01:26:21.726068 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:21.726066    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"
Jul 28 01:26:21.726301 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:21.726107    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885964179e9ede61214a186200dd1641658e5ef46974aadab36c8896bf64c4e1"
Jul 28 01:26:22.508184 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.508121    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:26:22.695849 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.695814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-socket\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.695849 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.695857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-podinfo\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.695929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-data\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.695989    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-certs\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696017    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp9nr\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-kube-api-access-fp9nr\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696042    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-credential-socket\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696088    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-envoy\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696129 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696127    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-token\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696424 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696123    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:26:22.696424 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:26:22.696424 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552257c-00a4-47bf-b995-d4b07cbafdd6-istiod-ca-cert\") pod \"1552257c-00a4-47bf-b995-d4b07cbafdd6\" (UID: \"1552257c-00a4-47bf-b995-d4b07cbafdd6\") "
Jul 28 01:26:22.696576 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696423    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:26:22.696714 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696690    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.696714 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696715    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.696900 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696719    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1552257c-00a4-47bf-b995-d4b07cbafdd6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:22.696900 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.696729    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.697345 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.697322    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-data" (OuterVolumeSpecName: "istio-data") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:26:22.698252 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.698220    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:26:22.698385 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.698350    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-token" (OuterVolumeSpecName: "istio-token") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:22.698655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.698629    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:26:22.698655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.698632    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-kube-api-access-fp9nr" (OuterVolumeSpecName: "kube-api-access-fp9nr") pod "1552257c-00a4-47bf-b995-d4b07cbafdd6" (UID: "1552257c-00a4-47bf-b995-d4b07cbafdd6"). InnerVolumeSpecName "kube-api-access-fp9nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:22.733140 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.733104    2564 generic.go:356] "Generic (PLEG): container finished" podID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerID="1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e" exitCode=0
Jul 28 01:26:22.733319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.733147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" event={"ID":"1552257c-00a4-47bf-b995-d4b07cbafdd6","Type":"ContainerDied","Data":"1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e"}
Jul 28 01:26:22.733319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.733151    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"
Jul 28 01:26:22.733319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.733190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq" event={"ID":"1552257c-00a4-47bf-b995-d4b07cbafdd6","Type":"ContainerDied","Data":"fa3f45c8fa3f04cc56a9d9e572d80eae271f0af15bcac21d9975d54151595a75"}
Jul 28 01:26:22.733319 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.733210    2564 scope.go:117] "RemoveContainer" containerID="1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e"
Jul 28 01:26:22.743408 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.743386    2564 scope.go:117] "RemoveContainer" containerID="d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"
Jul 28 01:26:22.752446 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.752425    2564 scope.go:117] "RemoveContainer" containerID="1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e"
Jul 28 01:26:22.752781 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:26:22.752760    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e\": container with ID starting with 1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e not found: ID does not exist" containerID="1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e"
Jul 28 01:26:22.752837 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.752793    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e"} err="failed to get container status \"1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e\": rpc error: code = NotFound desc = could not find container \"1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e\": container with ID starting with 1c0947986bb934f75a22ade4475b35308e13c0c54b631df3b145487a7f146f5e not found: ID does not exist"
Jul 28 01:26:22.752837 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.752813    2564 scope.go:117] "RemoveContainer" containerID="d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"
Jul 28 01:26:22.753101 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:26:22.753080    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3\": container with ID starting with d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3 not found: ID does not exist" containerID="d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"
Jul 28 01:26:22.753148 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.753105    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3"} err="failed to get container status \"d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3\": rpc error: code = NotFound desc = could not find container \"d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3\": container with ID starting with d7005f662e2f71ea16cb805a3868856cd9720b451bf8e7b687cc3303c3f6b6e3 not found: ID does not exist"
Jul 28 01:26:22.757785 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.757757    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"]
Jul 28 01:26:22.761796 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.761727    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5222f2-openshift-default-5bdb459b9d-dp7mq"]
Jul 28 01:26:22.797207 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.797166    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.797207 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.797198    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.797207 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.797212    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp9nr\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-kube-api-access-fp9nr\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.797453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.797225    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.797453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.797236    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552257c-00a4-47bf-b995-d4b07cbafdd6-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.797453 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.797246    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552257c-00a4-47bf-b995-d4b07cbafdd6-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:22.950212 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:22.950174    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" path="/var/lib/kubelet/pods/1552257c-00a4-47bf-b995-d4b07cbafdd6/volumes"
Jul 28 01:26:33.067331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.067244    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"]
Jul 28 01:26:33.067881 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.067501    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg" podUID="75ad4cc3-ac79-4ec5-897e-80af94996974" containerName="maas-api" containerID="cri-o://51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302" gracePeriod=30
Jul 28 01:26:33.322098 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.322032    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:26:33.380971 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.380901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ad4cc3-ac79-4ec5-897e-80af94996974-openshift-service-ca\") pod \"75ad4cc3-ac79-4ec5-897e-80af94996974\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") "
Jul 28 01:26:33.380971 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.380973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfzvc\" (UniqueName: \"kubernetes.io/projected/75ad4cc3-ac79-4ec5-897e-80af94996974-kube-api-access-bfzvc\") pod \"75ad4cc3-ac79-4ec5-897e-80af94996974\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") "
Jul 28 01:26:33.381239 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.381043    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ad4cc3-ac79-4ec5-897e-80af94996974-maas-api-tls\") pod \"75ad4cc3-ac79-4ec5-897e-80af94996974\" (UID: \"75ad4cc3-ac79-4ec5-897e-80af94996974\") "
Jul 28 01:26:33.381294 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.381273    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ad4cc3-ac79-4ec5-897e-80af94996974-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75ad4cc3-ac79-4ec5-897e-80af94996974" (UID: "75ad4cc3-ac79-4ec5-897e-80af94996974"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:33.383050 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.383022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ad4cc3-ac79-4ec5-897e-80af94996974-kube-api-access-bfzvc" (OuterVolumeSpecName: "kube-api-access-bfzvc") pod "75ad4cc3-ac79-4ec5-897e-80af94996974" (UID: "75ad4cc3-ac79-4ec5-897e-80af94996974"). InnerVolumeSpecName "kube-api-access-bfzvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:33.383167 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.383060    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ad4cc3-ac79-4ec5-897e-80af94996974-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75ad4cc3-ac79-4ec5-897e-80af94996974" (UID: "75ad4cc3-ac79-4ec5-897e-80af94996974"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:26:33.481819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.481780    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ad4cc3-ac79-4ec5-897e-80af94996974-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:33.481819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.481813    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzvc\" (UniqueName: \"kubernetes.io/projected/75ad4cc3-ac79-4ec5-897e-80af94996974-kube-api-access-bfzvc\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:33.481819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.481824    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ad4cc3-ac79-4ec5-897e-80af94996974-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:26:33.790736 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.790698    2564 generic.go:356] "Generic (PLEG): container finished" podID="75ad4cc3-ac79-4ec5-897e-80af94996974" containerID="51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302" exitCode=0
Jul 28 01:26:33.790965 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.790749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg" event={"ID":"75ad4cc3-ac79-4ec5-897e-80af94996974","Type":"ContainerDied","Data":"51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302"}
Jul 28 01:26:33.790965 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.790788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg" event={"ID":"75ad4cc3-ac79-4ec5-897e-80af94996974","Type":"ContainerDied","Data":"fd1e7600b576aa69471652cc2fb762fad8b308d0f620d32bd4b0b861496e5187"}
Jul 28 01:26:33.790965 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.790816    2564 scope.go:117] "RemoveContainer" containerID="51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302"
Jul 28 01:26:33.790965 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.790751    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"
Jul 28 01:26:33.800258 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.800240    2564 scope.go:117] "RemoveContainer" containerID="51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302"
Jul 28 01:26:33.800596 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:26:33.800575    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302\": container with ID starting with 51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302 not found: ID does not exist" containerID="51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302"
Jul 28 01:26:33.800642 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.800606    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302"} err="failed to get container status \"51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302\": rpc error: code = NotFound desc = could not find container \"51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302\": container with ID starting with 51c1d5cd5767817d4241019d05bf8ffbd31596e7f5c939e37d3f53eb954a6302 not found: ID does not exist"
Jul 28 01:26:33.812544 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.812517    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"]
Jul 28 01:26:33.816646 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:33.816624    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9c8614-587c94497b-9zrrg"]
Jul 28 01:26:34.944812 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:34.944782    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ad4cc3-ac79-4ec5-897e-80af94996974" path="/var/lib/kubelet/pods/75ad4cc3-ac79-4ec5-897e-80af94996974/volumes"
Jul 28 01:26:37.660329 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:37.660292    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"]
Jul 28 01:26:37.665514 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:37.665485    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9c8614-pfjfz"]
Jul 28 01:26:37.903370 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:37.903334    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:26:38.945112 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:38.945081    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745483bb-d5c8-451c-ad87-fa62407985c2" path="/var/lib/kubelet/pods/745483bb-d5c8-451c-ad87-fa62407985c2/volumes"
Jul 28 01:26:45.128329 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128292    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"]
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128803    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128826    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128838    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128848    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128875    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ad4cc3-ac79-4ec5-897e-80af94996974" containerName="maas-api"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128884    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ad4cc3-ac79-4ec5-897e-80af94996974" containerName="maas-api"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128897    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745483bb-d5c8-451c-ad87-fa62407985c2" containerName="revoke-keys"
Jul 28 01:26:45.128911 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.128905    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="745483bb-d5c8-451c-ad87-fa62407985c2" containerName="revoke-keys"
Jul 28 01:26:45.129362 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.129016    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy"
Jul 28 01:26:45.129362 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.129032    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ad4cc3-ac79-4ec5-897e-80af94996974" containerName="maas-api"
Jul 28 01:26:45.129362 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.129043    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="745483bb-d5c8-451c-ad87-fa62407985c2" containerName="revoke-keys"
Jul 28 01:26:45.133828 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.133797    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.136702 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.136679    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-14a1d0-openshift-default-dockercfg-6rrrd\""
Jul 28 01:26:45.146077 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.146046    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"]
Jul 28 01:26:45.184396 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-certs\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184576 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-podinfo\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184576 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-envoy\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184576 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghnv\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-kube-api-access-5ghnv\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184576 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-data\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184772 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-token\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184772 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istiod-ca-cert\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184772 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-socket\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.184772 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.184657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-credential-socket\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-socket\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-credential-socket\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286357 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-certs\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286357 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-podinfo\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286357 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-envoy\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286357 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghnv\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-kube-api-access-5ghnv\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286357 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-data\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-certs\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-credential-socket\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-socket\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.286739 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-token\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.287025 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istiod-ca-cert\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.287025 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.286876    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-data\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.287301 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.287283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istiod-ca-cert\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.289174 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.289155    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-envoy\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.289466 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.289443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-podinfo\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.294627 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.294599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-token\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.294993 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.294972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghnv\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-kube-api-access-5ghnv\") pod \"e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") " pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.446304 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.446269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:45.594629 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.594603    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"]
Jul 28 01:26:45.597267 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:26:45.597234    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358d73d2_54f3_4cb5_a13f_5c19dfd13142.slice/crio-feac699e8700e18127fa2b5c66fe61a9c413d96ec1460f6effeb27e773998a47 WatchSource:0}: Error finding container feac699e8700e18127fa2b5c66fe61a9c413d96ec1460f6effeb27e773998a47: Status 404 returned error can't find the container with id feac699e8700e18127fa2b5c66fe61a9c413d96ec1460f6effeb27e773998a47
Jul 28 01:26:45.599443 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.599408    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:26:45.599537 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.599490    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:26:45.599537 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.599530    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:26:45.847686 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.847591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" event={"ID":"358d73d2-54f3-4cb5-a13f-5c19dfd13142","Type":"ContainerStarted","Data":"f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"}
Jul 28 01:26:45.847686 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.847652    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" event={"ID":"358d73d2-54f3-4cb5-a13f-5c19dfd13142","Type":"ContainerStarted","Data":"feac699e8700e18127fa2b5c66fe61a9c413d96ec1460f6effeb27e773998a47"}
Jul 28 01:26:45.866998 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:45.866930    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podStartSLOduration=0.866909106 podStartE2EDuration="866.909106ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:45.865534207 +0000 UTC m=+2437.421037393" watchObservedRunningTime="2026-07-28 01:26:45.866909106 +0000 UTC m=+2437.422412265"
Jul 28 01:26:46.446777 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:46.446738    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:46.451770 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:46.451743    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:46.852890 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:46.852808    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:46.853734 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:46.853714    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:26:47.830065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.830030    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"]
Jul 28 01:26:47.830487 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.830443    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1552257c-00a4-47bf-b995-d4b07cbafdd6" containerName="istio-proxy"
Jul 28 01:26:47.833370 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.833342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:47.836335 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.836311    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-14a1d0-serving-cert\""
Jul 28 01:26:47.836335 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.836324    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 01:26:47.836543 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.836391    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:26:47.837070 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.837046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plcgg\""
Jul 28 01:26:47.847669 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.847640    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"]
Jul 28 01:26:47.909052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.909012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hm9m\" (UniqueName: \"kubernetes.io/projected/5a5951a4-d442-48a4-ab83-bdd529c9719a-kube-api-access-7hm9m\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:47.909267 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.909065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:47.909267 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:47.909172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5951a4-d442-48a4-ab83-bdd529c9719a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.010475 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.010432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5951a4-d442-48a4-ab83-bdd529c9719a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.010691 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.010509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hm9m\" (UniqueName: \"kubernetes.io/projected/5a5951a4-d442-48a4-ab83-bdd529c9719a-kube-api-access-7hm9m\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.010691 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.010545    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.010691 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:26:48.010654    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-serving-cert: secret "maas-api-e2e-shared-a-14a1d0-serving-cert" not found
Jul 28 01:26:48.010876 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:26:48.010727    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls podName:5a5951a4-d442-48a4-ab83-bdd529c9719a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:48.510706699 +0000 UTC m=+2440.066209837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls") pod "maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" (UID: "5a5951a4-d442-48a4-ab83-bdd529c9719a") : secret "maas-api-e2e-shared-a-14a1d0-serving-cert" not found
Jul 28 01:26:48.011247 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.011224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5951a4-d442-48a4-ab83-bdd529c9719a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.019292 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.019260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hm9m\" (UniqueName: \"kubernetes.io/projected/5a5951a4-d442-48a4-ab83-bdd529c9719a-kube-api-access-7hm9m\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.515160 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.515121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.517755 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.517719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls\") pod \"maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.745635 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.745594    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:48.892632 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:48.892606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"]
Jul 28 01:26:48.895161 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:26:48.895132    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5951a4_d442_48a4_ab83_bdd529c9719a.slice/crio-bf1965e9e94dbb5d8d24ef8b3da30e1025585add5fd1390097a5c82bbfcc0553 WatchSource:0}: Error finding container bf1965e9e94dbb5d8d24ef8b3da30e1025585add5fd1390097a5c82bbfcc0553: Status 404 returned error can't find the container with id bf1965e9e94dbb5d8d24ef8b3da30e1025585add5fd1390097a5c82bbfcc0553
Jul 28 01:26:49.870866 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:49.870774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" event={"ID":"5a5951a4-d442-48a4-ab83-bdd529c9719a","Type":"ContainerStarted","Data":"1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62"}
Jul 28 01:26:49.870866 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:49.870827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" event={"ID":"5a5951a4-d442-48a4-ab83-bdd529c9719a","Type":"ContainerStarted","Data":"bf1965e9e94dbb5d8d24ef8b3da30e1025585add5fd1390097a5c82bbfcc0553"}
Jul 28 01:26:49.870866 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:49.870851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:49.892790 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:49.892731    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" podStartSLOduration=2.321346684 podStartE2EDuration="2.892715965s" totalImagesPullingTime="571.369281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:49.892440295 +0000 UTC m=+2441.447943476" watchObservedRunningTime="2026-07-28 01:26:49.892715965 +0000 UTC m=+2441.448219124"
Jul 28 01:26:55.542836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.542798    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"]
Jul 28 01:26:55.546596 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.546572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.550520 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.550498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-3131e3-serving-cert\""
Jul 28 01:26:55.562611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.562576    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"]
Jul 28 01:26:55.682611 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.682576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcf1949-f79b-4002-88a6-034f00791a64-maas-api-tls\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.682788 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.682693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9r7x\" (UniqueName: \"kubernetes.io/projected/cdcf1949-f79b-4002-88a6-034f00791a64-kube-api-access-j9r7x\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.682788 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.682763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf1949-f79b-4002-88a6-034f00791a64-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.784227 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.784188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9r7x\" (UniqueName: \"kubernetes.io/projected/cdcf1949-f79b-4002-88a6-034f00791a64-kube-api-access-j9r7x\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.784440 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.784245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf1949-f79b-4002-88a6-034f00791a64-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.784440 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.784284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcf1949-f79b-4002-88a6-034f00791a64-maas-api-tls\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.785028 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.785002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf1949-f79b-4002-88a6-034f00791a64-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.786731 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.786711    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcf1949-f79b-4002-88a6-034f00791a64-maas-api-tls\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.793108 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.793049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9r7x\" (UniqueName: \"kubernetes.io/projected/cdcf1949-f79b-4002-88a6-034f00791a64-kube-api-access-j9r7x\") pod \"maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.856394 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.856347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:55.883030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:55.883002    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:26:56.011151 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:56.011116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"]
Jul 28 01:26:56.015365 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:26:56.015330    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcf1949_f79b_4002_88a6_034f00791a64.slice/crio-457f7de68f0803167484218dcb77d0427d73a9fbc62c925f2c5c2fb700a97b3b WatchSource:0}: Error finding container 457f7de68f0803167484218dcb77d0427d73a9fbc62c925f2c5c2fb700a97b3b: Status 404 returned error can't find the container with id 457f7de68f0803167484218dcb77d0427d73a9fbc62c925f2c5c2fb700a97b3b
Jul 28 01:26:56.545555 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:56.545511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:26:56.910768 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:56.910729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh" event={"ID":"cdcf1949-f79b-4002-88a6-034f00791a64","Type":"ContainerStarted","Data":"c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418"}
Jul 28 01:26:56.910990 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:56.910785    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh" event={"ID":"cdcf1949-f79b-4002-88a6-034f00791a64","Type":"ContainerStarted","Data":"457f7de68f0803167484218dcb77d0427d73a9fbc62c925f2c5c2fb700a97b3b"}
Jul 28 01:26:56.910990 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:56.910829    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:26:56.928265 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:26:56.928204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh" podStartSLOduration=1.572328074 podStartE2EDuration="1.928188927s" totalImagesPullingTime="355.860853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:56.92625728 +0000 UTC m=+2448.481760461" watchObservedRunningTime="2026-07-28 01:26:56.928188927 +0000 UTC m=+2448.483692087"
Jul 28 01:27:02.895145 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:02.895092    2564 patch_prober.go:28] interesting pod/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 01:27:02.895632 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:02.895175    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 01:27:02.922887 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:02.922856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:27:04.955730 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:04.955647    2564 generic.go:356] "Generic (PLEG): container finished" podID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerID="f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92" exitCode=0
Jul 28 01:27:04.955730 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:04.955721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" event={"ID":"358d73d2-54f3-4cb5-a13f-5c19dfd13142","Type":"ContainerDied","Data":"f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"}
Jul 28 01:27:04.956188 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:04.956174    2564 scope.go:117] "RemoveContainer" containerID="f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"
Jul 28 01:27:04.957178 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:04.957149    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:27:04.957236 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:04.957216    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:27:04.957273 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:04.957245    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:27:05.964530 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:05.964485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" event={"ID":"358d73d2-54f3-4cb5-a13f-5c19dfd13142","Type":"ContainerStarted","Data":"28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f"}
Jul 28 01:27:06.447208 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:06.447163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:27:06.447451 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:06.447430    2564 patch_prober.go:28] interesting pod/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 01:27:06.447502 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:06.447476    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 01:27:07.447048 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:07.447002    2564 patch_prober.go:28] interesting pod/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 01:27:07.447589 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:07.447076    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 01:27:07.714841 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:07.714752    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:27:08.447115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:08.447073    2564 patch_prober.go:28] interesting pod/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 01:27:08.447550 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:08.447142    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 01:27:09.243818 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:09.243786    2564 scope.go:117] "RemoveContainer" containerID="af4ffe468508ffaa1f87d9879e48cab1f8de3f96f553e62e91187ad8117e902a"
Jul 28 01:27:09.446914 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:09.446879    2564 patch_prober.go:28] interesting pod/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 01:27:09.447116 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:09.446960    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 01:27:10.450786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:10.450752    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:27:10.451253 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:10.451161    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:27:10.451892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:10.451870    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:27:33.300980 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:27:33.300927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:28:21.001956 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:28:21.001832    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:28:47.105435 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:28:47.105392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:29:13.205287 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:29:13.205245    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:29:29.904979 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:29:29.904921    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:29:52.206192 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:29:52.206152    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:30:10.198459 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:30:10.198418    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:30:32.098075 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:30:32.098035    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:30:48.904023 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:30:48.903980    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:31:01.322487 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:01.322395    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:31:09.087504 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:09.087474    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:31:09.102068 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:09.102039    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:31:09.302012 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:09.301982    2564 scope.go:117] "RemoveContainer" containerID="db89f84fc621682f68de4f4e598351284c03d06c68894db7d2121ebad6739d46"
Jul 28 01:31:09.310474 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:09.310452    2564 scope.go:117] "RemoveContainer" containerID="56484b09fe3926eb79647e54d2ff6c596e9879428e50089180a8fcea5eeae16a"
Jul 28 01:31:23.103984 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:23.103928    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:31:39.010587 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:31:39.010551    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:32:01.510294 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:32:01.510259    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:32:18.906098 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:32:18.906063    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:32:46.416114 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:32:46.416019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:33:02.206332 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:33:02.206297    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:33:09.341022 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:33:09.340988    2564 scope.go:117] "RemoveContainer" containerID="b10991e2aadc2c030eaf2cbd07920fd14c35942b40531aac08a91f4170a81a9c"
Jul 28 01:33:11.400002 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:33:11.399968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:33:30.909277 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:33:30.909236    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:34:03.507836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:34:03.507743    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:34:21.399042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:34:21.399007    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:34:41.120883 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:34:41.120834    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:34:59.104569 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:34:59.104529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:35:27.016135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:35:27.016090    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:36:01.803868 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:01.803821    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:36:09.118341 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:09.118310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:36:09.138501 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:09.138473    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:36:16.007335 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:16.007282    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:36:30.701722 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:30.701674    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:36:44.004523 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:44.004466    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:36:57.224683 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.224638    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"]
Jul 28 01:36:57.228786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.228762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.231493 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.231462    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c0266d-openshift-default-dockercfg-45kzm\""
Jul 28 01:36:57.240143 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.240109    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"]
Jul 28 01:36:57.371390 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371338    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-podinfo\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371390 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-credential-socket\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-token\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-socket\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-data\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-envoy\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371655 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-certs\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371841 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68750271-f3fe-4691-8422-ca51f2b2a93a-istiod-ca-cert\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.371841 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.371702    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jwf\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-kube-api-access-l9jwf\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472420 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-token\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-socket\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-data\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-envoy\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-certs\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68750271-f3fe-4691-8422-ca51f2b2a93a-istiod-ca-cert\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jwf\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-kube-api-access-l9jwf\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-podinfo\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.472650 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-credential-socket\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.473125 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-socket\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.473125 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.472929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-certs\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.473125 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.473078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-credential-socket\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.473281 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.473235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-data\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.473348 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.473301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68750271-f3fe-4691-8422-ca51f2b2a93a-istiod-ca-cert\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.475161 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.475089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-envoy\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.475459 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.475435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-podinfo\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.480484 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.480448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-token\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.480597 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.480531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jwf\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-kube-api-access-l9jwf\") pod \"e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") " pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.544073 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.544031    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:57.697585 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.697556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"]
Jul 28 01:36:57.700318 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:36:57.700285    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68750271_f3fe_4691_8422_ca51f2b2a93a.slice/crio-9a5e08212527ab04aac109dff70074918f9fe6ea5f60afcfb1dcd4dcbef6bdcc WatchSource:0}: Error finding container 9a5e08212527ab04aac109dff70074918f9fe6ea5f60afcfb1dcd4dcbef6bdcc: Status 404 returned error can't find the container with id 9a5e08212527ab04aac109dff70074918f9fe6ea5f60afcfb1dcd4dcbef6bdcc
Jul 28 01:36:57.703057 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.703015    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:36:57.703172 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.703112    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:36:57.703172 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.703155    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:36:57.900110 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.900064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" event={"ID":"68750271-f3fe-4691-8422-ca51f2b2a93a","Type":"ContainerStarted","Data":"2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"}
Jul 28 01:36:57.900282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.900147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" event={"ID":"68750271-f3fe-4691-8422-ca51f2b2a93a","Type":"ContainerStarted","Data":"9a5e08212527ab04aac109dff70074918f9fe6ea5f60afcfb1dcd4dcbef6bdcc"}
Jul 28 01:36:57.919694 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:57.919622    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podStartSLOduration=0.919598584 podStartE2EDuration="919.598584ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:57.918071268 +0000 UTC m=+3049.473574453" watchObservedRunningTime="2026-07-28 01:36:57.919598584 +0000 UTC m=+3049.475101744"
Jul 28 01:36:58.545189 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:58.545141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:58.551458 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:58.551426    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:58.905722 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:58.905618    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:36:58.906587 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:36:58.906567    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:37:00.017438 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.017395    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"]
Jul 28 01:37:00.021780 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.021747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.024193 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.024163    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c0266d-serving-cert\""
Jul 28 01:37:00.030558 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.030522    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"]
Jul 28 01:37:00.199064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.199025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.199295 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.199139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980e53f3-7f8f-4884-89f8-160e1a62d27b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.199295 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.199214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p595\" (UniqueName: \"kubernetes.io/projected/980e53f3-7f8f-4884-89f8-160e1a62d27b-kube-api-access-5p595\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.300459 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.300339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p595\" (UniqueName: \"kubernetes.io/projected/980e53f3-7f8f-4884-89f8-160e1a62d27b-kube-api-access-5p595\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.300459 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.300400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.300730 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.300467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980e53f3-7f8f-4884-89f8-160e1a62d27b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.300730 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:37:00.300556    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-serving-cert: secret "maas-api-e2e-ipp-a-c0266d-serving-cert" not found
Jul 28 01:37:00.300730 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:37:00.300642    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls podName:980e53f3-7f8f-4884-89f8-160e1a62d27b nodeName:}" failed. No retries permitted until 2026-07-28 01:37:00.800618283 +0000 UTC m=+3052.356121442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls") pod "maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" (UID: "980e53f3-7f8f-4884-89f8-160e1a62d27b") : secret "maas-api-e2e-ipp-a-c0266d-serving-cert" not found
Jul 28 01:37:00.301130 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.301108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980e53f3-7f8f-4884-89f8-160e1a62d27b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.313110 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.313066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p595\" (UniqueName: \"kubernetes.io/projected/980e53f3-7f8f-4884-89f8-160e1a62d27b-kube-api-access-5p595\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.804947 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.804901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.807614 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.807569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:00.936213 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:00.936168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:01.090233 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:01.090206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"]
Jul 28 01:37:01.092111 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:37:01.092077    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980e53f3_7f8f_4884_89f8_160e1a62d27b.slice/crio-2ce1f94a0a27230852722be33d759c6aae1e0b258b3225e68356b0a80445f054 WatchSource:0}: Error finding container 2ce1f94a0a27230852722be33d759c6aae1e0b258b3225e68356b0a80445f054: Status 404 returned error can't find the container with id 2ce1f94a0a27230852722be33d759c6aae1e0b258b3225e68356b0a80445f054
Jul 28 01:37:01.093550 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:01.093526    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:37:01.924040 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:01.923904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" event={"ID":"980e53f3-7f8f-4884-89f8-160e1a62d27b","Type":"ContainerStarted","Data":"cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb"}
Jul 28 01:37:01.924250 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:01.924043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" event={"ID":"980e53f3-7f8f-4884-89f8-160e1a62d27b","Type":"ContainerStarted","Data":"2ce1f94a0a27230852722be33d759c6aae1e0b258b3225e68356b0a80445f054"}
Jul 28 01:37:01.924250 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:01.924078    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:01.941109 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:01.941051    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" podStartSLOduration=2.376698658 podStartE2EDuration="2.941033571s" totalImagesPullingTime="564.334913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:01.93977749 +0000 UTC m=+3053.495280687" watchObservedRunningTime="2026-07-28 01:37:01.941033571 +0000 UTC m=+3053.496536730"
Jul 28 01:37:07.813602 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:07.813563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:37:07.935593 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:07.935561    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:37:14.948139 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:14.948096    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 01:37:14.948570 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:14.948176    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 01:37:17.018625 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:17.018590    2564 generic.go:356] "Generic (PLEG): container finished" podID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerID="2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da" exitCode=0
Jul 28 01:37:17.019128 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:17.018669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" event={"ID":"68750271-f3fe-4691-8422-ca51f2b2a93a","Type":"ContainerDied","Data":"2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"}
Jul 28 01:37:17.019176 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:17.019129    2564 scope.go:117] "RemoveContainer" containerID="2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"
Jul 28 01:37:17.020314 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:17.020279    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:37:17.020432 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:17.020364    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:37:17.020432 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:17.020399    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 01:37:18.026188 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.026153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" event={"ID":"68750271-f3fe-4691-8422-ca51f2b2a93a","Type":"ContainerStarted","Data":"65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb"}
Jul 28 01:37:18.256173 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.256132    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"]
Jul 28 01:37:18.258742 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.258715    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.271962 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.264258    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1ce59a-serving-cert\""
Jul 28 01:37:18.292022 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.290047    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"]
Jul 28 01:37:18.376809 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.376758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmqq\" (UniqueName: \"kubernetes.io/projected/cac2a642-5a1a-4b88-b989-e6a25350cec7-kube-api-access-jdmqq\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.377263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.377235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.377445 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.377430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac2a642-5a1a-4b88-b989-e6a25350cec7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.479134 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.479084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.479597 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.479570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac2a642-5a1a-4b88-b989-e6a25350cec7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.480766 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.480738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmqq\" (UniqueName: \"kubernetes.io/projected/cac2a642-5a1a-4b88-b989-e6a25350cec7-kube-api-access-jdmqq\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.481386 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:37:18.479507    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-serving-cert: secret "maas-api-e2e-ipp-b-1ce59a-serving-cert" not found
Jul 28 01:37:18.481597 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:37:18.481584    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls podName:cac2a642-5a1a-4b88-b989-e6a25350cec7 nodeName:}" failed. No retries permitted until 2026-07-28 01:37:18.981558743 +0000 UTC m=+3070.537061884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls") pod "maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" (UID: "cac2a642-5a1a-4b88-b989-e6a25350cec7") : secret "maas-api-e2e-ipp-b-1ce59a-serving-cert" not found
Jul 28 01:37:18.481985 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.480544    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac2a642-5a1a-4b88-b989-e6a25350cec7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.491157 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.491124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmqq\" (UniqueName: \"kubernetes.io/projected/cac2a642-5a1a-4b88-b989-e6a25350cec7-kube-api-access-jdmqq\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.545153 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.545051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:37:18.545331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.545298    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 01:37:18.545390 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.545368    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 01:37:18.990959 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.990793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:18.995751 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:18.995678    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:19.212586 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:19.212545    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:19.433540 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:19.433511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"]
Jul 28 01:37:19.545398 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:19.545352    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 01:37:19.545586 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:19.545433    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 01:37:20.038728 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:20.038693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" event={"ID":"cac2a642-5a1a-4b88-b989-e6a25350cec7","Type":"ContainerStarted","Data":"82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3"}
Jul 28 01:37:20.039041 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:20.039016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" event={"ID":"cac2a642-5a1a-4b88-b989-e6a25350cec7","Type":"ContainerStarted","Data":"3dbc01ee4a6b743739eb65dfd6f63ec144b5618c8f94e59c85e3c5601b617073"}
Jul 28 01:37:20.039149 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:20.039118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:20.055578 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:20.055509    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" podStartSLOduration=1.6818759330000002 podStartE2EDuration="2.05548652s" totalImagesPullingTime="373.610587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:20.053800422 +0000 UTC m=+3071.609303592" watchObservedRunningTime="2026-07-28 01:37:20.05548652 +0000 UTC m=+3071.610989682"
Jul 28 01:37:20.545160 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:20.545123    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 01:37:20.545674 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:20.545645    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 01:37:21.545467 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:21.545422    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 01:37:21.545920 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:21.545498    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 01:37:22.006793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:22.006750    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:37:22.549511 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:22.549472    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:37:22.550022 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:22.549839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:37:22.550866 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:22.550846    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:37:26.049830 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:26.049801    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:37:56.514455 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:37:56.514387    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:38:24.913867 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:24.913824    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:38:41.607712 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.607616    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"]
Jul 28 01:38:41.612173 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.612146    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.614510 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.614485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-5ecc77-serving-cert\""
Jul 28 01:38:41.622791 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.622752    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"]
Jul 28 01:38:41.769316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.769264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.769316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.769324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1859-4633-400b-b7b3-f9dedd975597-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.769589 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.769463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784vn\" (UniqueName: \"kubernetes.io/projected/9e5f1859-4633-400b-b7b3-f9dedd975597-kube-api-access-784vn\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.870760 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.870642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784vn\" (UniqueName: \"kubernetes.io/projected/9e5f1859-4633-400b-b7b3-f9dedd975597-kube-api-access-784vn\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.870760 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.870752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.871079 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.870800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1859-4633-400b-b7b3-f9dedd975597-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.871079 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:38:41.870892    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-serving-cert: secret "maas-api-e2e-ipp-cleanup-5ecc77-serving-cert" not found
Jul 28 01:38:41.871079 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:38:41.871010    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls podName:9e5f1859-4633-400b-b7b3-f9dedd975597 nodeName:}" failed. No retries permitted until 2026-07-28 01:38:42.370990631 +0000 UTC m=+3153.926493787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" (UID: "9e5f1859-4633-400b-b7b3-f9dedd975597") : secret "maas-api-e2e-ipp-cleanup-5ecc77-serving-cert" not found
Jul 28 01:38:41.871547 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.871523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1859-4633-400b-b7b3-f9dedd975597-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:41.880298 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:41.880269    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-784vn\" (UniqueName: \"kubernetes.io/projected/9e5f1859-4633-400b-b7b3-f9dedd975597-kube-api-access-784vn\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:42.376600 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:42.376551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:42.379189 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:42.379165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:42.525872 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:42.525830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:42.676366 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:42.676339    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"]
Jul 28 01:38:42.679101 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:38:42.679067    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5f1859_4633_400b_b7b3_f9dedd975597.slice/crio-88b5f780d4a328dcb94fa1276e3642812b5aa76c77f2c3680f3b90dbbd24f090 WatchSource:0}: Error finding container 88b5f780d4a328dcb94fa1276e3642812b5aa76c77f2c3680f3b90dbbd24f090: Status 404 returned error can't find the container with id 88b5f780d4a328dcb94fa1276e3642812b5aa76c77f2c3680f3b90dbbd24f090
Jul 28 01:38:43.484427 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:43.484387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" event={"ID":"9e5f1859-4633-400b-b7b3-f9dedd975597","Type":"ContainerStarted","Data":"dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56"}
Jul 28 01:38:43.484649 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:43.484445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" event={"ID":"9e5f1859-4633-400b-b7b3-f9dedd975597","Type":"ContainerStarted","Data":"88b5f780d4a328dcb94fa1276e3642812b5aa76c77f2c3680f3b90dbbd24f090"}
Jul 28 01:38:43.484649 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:43.484518    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:43.503524 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:43.503459    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" podStartSLOduration=2.096384357 podStartE2EDuration="2.503440793s" totalImagesPullingTime="407.056436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:43.501483338 +0000 UTC m=+3155.056986512" watchObservedRunningTime="2026-07-28 01:38:43.503440793 +0000 UTC m=+3155.058943953"
Jul 28 01:38:49.419746 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:49.419700    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:38:49.497819 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:49.497777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:38:55.962605 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:55.962569    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"]
Jul 28 01:38:55.966326 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:55.966301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:55.968997 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:55.968966    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:38:55.979327 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:55.979291    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"]
Jul 28 01:38:55.995429 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:55.995383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56765104-d2ef-4add-bb51-9fdb99f8cbed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd\" (UID: \"56765104-d2ef-4add-bb51-9fdb99f8cbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:56.096502 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:56.096457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56765104-d2ef-4add-bb51-9fdb99f8cbed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd\" (UID: \"56765104-d2ef-4add-bb51-9fdb99f8cbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:56.097181 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:56.097153    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56765104-d2ef-4add-bb51-9fdb99f8cbed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd\" (UID: \"56765104-d2ef-4add-bb51-9fdb99f8cbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:56.276893 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:56.276794    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:56.424177 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:56.424134    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"]
Jul 28 01:38:56.425062 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:38:56.425035    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56765104_d2ef_4add_bb51_9fdb99f8cbed.slice/crio-ce7e7c5acb75f85d695dd9a99101a0049ab5a4c1326e03f614c2e2464361fa32 WatchSource:0}: Error finding container ce7e7c5acb75f85d695dd9a99101a0049ab5a4c1326e03f614c2e2464361fa32: Status 404 returned error can't find the container with id ce7e7c5acb75f85d695dd9a99101a0049ab5a4c1326e03f614c2e2464361fa32
Jul 28 01:38:56.555305 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:56.555214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd" event={"ID":"56765104-d2ef-4add-bb51-9fdb99f8cbed","Type":"ContainerStarted","Data":"ce7e7c5acb75f85d695dd9a99101a0049ab5a4c1326e03f614c2e2464361fa32"}
Jul 28 01:38:57.562640 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:57.562604    2564 generic.go:356] "Generic (PLEG): container finished" podID="56765104-d2ef-4add-bb51-9fdb99f8cbed" containerID="aebefa7ece6a575342f78282345d21087152b1bcdba0d73de5be2a4f48365046" exitCode=0
Jul 28 01:38:57.563100 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:57.562693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd" event={"ID":"56765104-d2ef-4add-bb51-9fdb99f8cbed","Type":"ContainerDied","Data":"aebefa7ece6a575342f78282345d21087152b1bcdba0d73de5be2a4f48365046"}
Jul 28 01:38:58.705793 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:58.705759    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:58.718894 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:58.718863    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56765104-d2ef-4add-bb51-9fdb99f8cbed-maas-api-service-ca\") pod \"56765104-d2ef-4add-bb51-9fdb99f8cbed\" (UID: \"56765104-d2ef-4add-bb51-9fdb99f8cbed\") "
Jul 28 01:38:58.719269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:58.719245    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56765104-d2ef-4add-bb51-9fdb99f8cbed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56765104-d2ef-4add-bb51-9fdb99f8cbed" (UID: "56765104-d2ef-4add-bb51-9fdb99f8cbed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:58.820024 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:58.819896    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56765104-d2ef-4add-bb51-9fdb99f8cbed-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:38:59.574431 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:59.574323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd" event={"ID":"56765104-d2ef-4add-bb51-9fdb99f8cbed","Type":"ContainerDied","Data":"ce7e7c5acb75f85d695dd9a99101a0049ab5a4c1326e03f614c2e2464361fa32"}
Jul 28 01:38:59.574431 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:59.574356    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"
Jul 28 01:38:59.574431 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:38:59.574422    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7e7c5acb75f85d695dd9a99101a0049ab5a4c1326e03f614c2e2464361fa32"
Jul 28 01:39:07.332954 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.332894    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"]
Jul 28 01:39:07.333498 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.333156    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" podUID="9e5f1859-4633-400b-b7b3-f9dedd975597" containerName="maas-api" containerID="cri-o://dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56" gracePeriod=30
Jul 28 01:39:07.611624 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.611593    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:39:07.616534 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.616491    2564 generic.go:356] "Generic (PLEG): container finished" podID="9e5f1859-4633-400b-b7b3-f9dedd975597" containerID="dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56" exitCode=0
Jul 28 01:39:07.616740 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.616545    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"
Jul 28 01:39:07.616740 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.616575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" event={"ID":"9e5f1859-4633-400b-b7b3-f9dedd975597","Type":"ContainerDied","Data":"dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56"}
Jul 28 01:39:07.616740 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.616656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw" event={"ID":"9e5f1859-4633-400b-b7b3-f9dedd975597","Type":"ContainerDied","Data":"88b5f780d4a328dcb94fa1276e3642812b5aa76c77f2c3680f3b90dbbd24f090"}
Jul 28 01:39:07.616740 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.616681    2564 scope.go:117] "RemoveContainer" containerID="dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56"
Jul 28 01:39:07.626243 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.626217    2564 scope.go:117] "RemoveContainer" containerID="dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56"
Jul 28 01:39:07.626598 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:39:07.626575    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56\": container with ID starting with dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56 not found: ID does not exist" containerID="dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56"
Jul 28 01:39:07.626669 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.626610    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56"} err="failed to get container status \"dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56\": rpc error: code = NotFound desc = could not find container \"dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56\": container with ID starting with dab06c38c50c8ad62d9a144baebbf70e2fbee94b3b54598a15bbaff6e3296e56 not found: ID does not exist"
Jul 28 01:39:07.699291 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.699236    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-784vn\" (UniqueName: \"kubernetes.io/projected/9e5f1859-4633-400b-b7b3-f9dedd975597-kube-api-access-784vn\") pod \"9e5f1859-4633-400b-b7b3-f9dedd975597\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") "
Jul 28 01:39:07.699506 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.699300    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls\") pod \"9e5f1859-4633-400b-b7b3-f9dedd975597\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") "
Jul 28 01:39:07.699506 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.699408    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1859-4633-400b-b7b3-f9dedd975597-openshift-service-ca\") pod \"9e5f1859-4633-400b-b7b3-f9dedd975597\" (UID: \"9e5f1859-4633-400b-b7b3-f9dedd975597\") "
Jul 28 01:39:07.699799 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.699769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5f1859-4633-400b-b7b3-f9dedd975597-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e5f1859-4633-400b-b7b3-f9dedd975597" (UID: "9e5f1859-4633-400b-b7b3-f9dedd975597"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:07.701484 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.701457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e5f1859-4633-400b-b7b3-f9dedd975597" (UID: "9e5f1859-4633-400b-b7b3-f9dedd975597"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:07.701921 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.701899    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5f1859-4633-400b-b7b3-f9dedd975597-kube-api-access-784vn" (OuterVolumeSpecName: "kube-api-access-784vn") pod "9e5f1859-4633-400b-b7b3-f9dedd975597" (UID: "9e5f1859-4633-400b-b7b3-f9dedd975597"). InnerVolumeSpecName "kube-api-access-784vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:07.800745 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.800701    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1859-4633-400b-b7b3-f9dedd975597-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:07.800745 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.800741    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-784vn\" (UniqueName: \"kubernetes.io/projected/9e5f1859-4633-400b-b7b3-f9dedd975597-kube-api-access-784vn\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:07.800745 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.800754    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1859-4633-400b-b7b3-f9dedd975597-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:07.964115 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.964071    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"]
Jul 28 01:39:07.979540 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:07.979505    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ecc77-85c9cbf686-tlcvw"]
Jul 28 01:39:08.947267 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:08.947232    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5f1859-4633-400b-b7b3-f9dedd975597" path="/var/lib/kubelet/pods/9e5f1859-4633-400b-b7b3-f9dedd975597/volumes"
Jul 28 01:39:11.117980 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:11.117945    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"]
Jul 28 01:39:11.120620 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:11.120586    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ecc77-r8rrd"]
Jul 28 01:39:12.018352 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:12.018313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:39:12.945325 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:12.945288    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56765104-d2ef-4add-bb51-9fdb99f8cbed" path="/var/lib/kubelet/pods/56765104-d2ef-4add-bb51-9fdb99f8cbed/volumes"
Jul 28 01:39:21.934905 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.934864    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"]
Jul 28 01:39:21.935343 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.935326    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5f1859-4633-400b-b7b3-f9dedd975597" containerName="maas-api"
Jul 28 01:39:21.935405 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.935346    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5f1859-4633-400b-b7b3-f9dedd975597" containerName="maas-api"
Jul 28 01:39:21.935405 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.935369    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56765104-d2ef-4add-bb51-9fdb99f8cbed" containerName="revoke-keys"
Jul 28 01:39:21.935405 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.935375    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="56765104-d2ef-4add-bb51-9fdb99f8cbed" containerName="revoke-keys"
Jul 28 01:39:21.935508 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.935470    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="56765104-d2ef-4add-bb51-9fdb99f8cbed" containerName="revoke-keys"
Jul 28 01:39:21.935508 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.935480    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5f1859-4633-400b-b7b3-f9dedd975597" containerName="maas-api"
Jul 28 01:39:21.938337 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.938320    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:21.940501 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.940479    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:39:21.944827 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:21.944800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"]
Jul 28 01:39:22.040851 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.040802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dcc316-6e22-443b-abe6-5b1b7922c582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp\" (UID: \"c9dcc316-6e22-443b-abe6-5b1b7922c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:22.141905 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.141863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dcc316-6e22-443b-abe6-5b1b7922c582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp\" (UID: \"c9dcc316-6e22-443b-abe6-5b1b7922c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:22.142559 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.142536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dcc316-6e22-443b-abe6-5b1b7922c582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp\" (UID: \"c9dcc316-6e22-443b-abe6-5b1b7922c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:22.251996 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.251860    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:22.397694 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.397653    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"]
Jul 28 01:39:22.399792 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:39:22.399758    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dcc316_6e22_443b_abe6_5b1b7922c582.slice/crio-00fbbeeceb0d36e58e0d092c545d402b2866fa3bee5e9395fd05453392ffc50c WatchSource:0}: Error finding container 00fbbeeceb0d36e58e0d092c545d402b2866fa3bee5e9395fd05453392ffc50c: Status 404 returned error can't find the container with id 00fbbeeceb0d36e58e0d092c545d402b2866fa3bee5e9395fd05453392ffc50c
Jul 28 01:39:22.700971 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.700918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp" event={"ID":"c9dcc316-6e22-443b-abe6-5b1b7922c582","Type":"ContainerStarted","Data":"d97f074e22e6a366cb5bb683f257805e93f1162b3e5cc72a5cc11f6fb11f6cbd"}
Jul 28 01:39:22.701160 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.701001    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp" event={"ID":"c9dcc316-6e22-443b-abe6-5b1b7922c582","Type":"ContainerStarted","Data":"00fbbeeceb0d36e58e0d092c545d402b2866fa3bee5e9395fd05453392ffc50c"}
Jul 28 01:39:22.715738 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:22.715678    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp" podStartSLOduration=1.715659167 podStartE2EDuration="1.715659167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:22.714558085 +0000 UTC m=+3194.270061246" watchObservedRunningTime="2026-07-28 01:39:22.715659167 +0000 UTC m=+3194.271162327"
Jul 28 01:39:23.708378 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:23.708341    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9dcc316-6e22-443b-abe6-5b1b7922c582" containerID="d97f074e22e6a366cb5bb683f257805e93f1162b3e5cc72a5cc11f6fb11f6cbd" exitCode=0
Jul 28 01:39:23.708836 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:23.708431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp" event={"ID":"c9dcc316-6e22-443b-abe6-5b1b7922c582","Type":"ContainerDied","Data":"d97f074e22e6a366cb5bb683f257805e93f1162b3e5cc72a5cc11f6fb11f6cbd"}
Jul 28 01:39:24.846145 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:24.846118    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:24.971010 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:24.970891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dcc316-6e22-443b-abe6-5b1b7922c582-maas-api-service-ca\") pod \"c9dcc316-6e22-443b-abe6-5b1b7922c582\" (UID: \"c9dcc316-6e22-443b-abe6-5b1b7922c582\") "
Jul 28 01:39:24.971280 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:24.971258    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9dcc316-6e22-443b-abe6-5b1b7922c582-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9dcc316-6e22-443b-abe6-5b1b7922c582" (UID: "c9dcc316-6e22-443b-abe6-5b1b7922c582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:25.073076 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:25.073033    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dcc316-6e22-443b-abe6-5b1b7922c582-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:25.720203 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:25.720170    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"
Jul 28 01:39:25.720203 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:25.720184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp" event={"ID":"c9dcc316-6e22-443b-abe6-5b1b7922c582","Type":"ContainerDied","Data":"00fbbeeceb0d36e58e0d092c545d402b2866fa3bee5e9395fd05453392ffc50c"}
Jul 28 01:39:25.720425 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:25.720238    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00fbbeeceb0d36e58e0d092c545d402b2866fa3bee5e9395fd05453392ffc50c"
Jul 28 01:39:37.506979 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:37.506914    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:39:39.733847 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:39.733811    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"]
Jul 28 01:39:39.734259 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:39.734079    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" podUID="980e53f3-7f8f-4884-89f8-160e1a62d27b" containerName="maas-api" containerID="cri-o://cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb" gracePeriod=30
Jul 28 01:39:40.002277 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.002207    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:39:40.015590 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.015563    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls\") pod \"980e53f3-7f8f-4884-89f8-160e1a62d27b\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") "
Jul 28 01:39:40.015724 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.015604    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p595\" (UniqueName: \"kubernetes.io/projected/980e53f3-7f8f-4884-89f8-160e1a62d27b-kube-api-access-5p595\") pod \"980e53f3-7f8f-4884-89f8-160e1a62d27b\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") "
Jul 28 01:39:40.015724 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.015644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980e53f3-7f8f-4884-89f8-160e1a62d27b-openshift-service-ca\") pod \"980e53f3-7f8f-4884-89f8-160e1a62d27b\" (UID: \"980e53f3-7f8f-4884-89f8-160e1a62d27b\") "
Jul 28 01:39:40.016065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.016037    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980e53f3-7f8f-4884-89f8-160e1a62d27b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "980e53f3-7f8f-4884-89f8-160e1a62d27b" (UID: "980e53f3-7f8f-4884-89f8-160e1a62d27b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:40.017603 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.017576    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "980e53f3-7f8f-4884-89f8-160e1a62d27b" (UID: "980e53f3-7f8f-4884-89f8-160e1a62d27b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:40.017722 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.017640    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980e53f3-7f8f-4884-89f8-160e1a62d27b-kube-api-access-5p595" (OuterVolumeSpecName: "kube-api-access-5p595") pod "980e53f3-7f8f-4884-89f8-160e1a62d27b" (UID: "980e53f3-7f8f-4884-89f8-160e1a62d27b"). InnerVolumeSpecName "kube-api-access-5p595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:40.117065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.117033    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980e53f3-7f8f-4884-89f8-160e1a62d27b-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.117065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.117061    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p595\" (UniqueName: \"kubernetes.io/projected/980e53f3-7f8f-4884-89f8-160e1a62d27b-kube-api-access-5p595\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.117065 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.117070    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980e53f3-7f8f-4884-89f8-160e1a62d27b-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.797528 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.797494    2564 generic.go:356] "Generic (PLEG): container finished" podID="980e53f3-7f8f-4884-89f8-160e1a62d27b" containerID="cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb" exitCode=0
Jul 28 01:39:40.798052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.797541    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"
Jul 28 01:39:40.798052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.797563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" event={"ID":"980e53f3-7f8f-4884-89f8-160e1a62d27b","Type":"ContainerDied","Data":"cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb"}
Jul 28 01:39:40.798052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.797608    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb" event={"ID":"980e53f3-7f8f-4884-89f8-160e1a62d27b","Type":"ContainerDied","Data":"2ce1f94a0a27230852722be33d759c6aae1e0b258b3225e68356b0a80445f054"}
Jul 28 01:39:40.798052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.797631    2564 scope.go:117] "RemoveContainer" containerID="cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb"
Jul 28 01:39:40.807068 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.807029    2564 scope.go:117] "RemoveContainer" containerID="cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb"
Jul 28 01:39:40.807352 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:39:40.807328    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb\": container with ID starting with cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb not found: ID does not exist" containerID="cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb"
Jul 28 01:39:40.807457 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.807358    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb"} err="failed to get container status \"cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb\": rpc error: code = NotFound desc = could not find container \"cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb\": container with ID starting with cfd892e26831f70976d31587566143181c442a1005df6bb2510a5d44d2bf55eb not found: ID does not exist"
Jul 28 01:39:40.818775 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.818742    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"]
Jul 28 01:39:40.824782 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.824760    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c0266d-6b4dd6465b-d8lpb"]
Jul 28 01:39:40.945304 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:40.945274    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980e53f3-7f8f-4884-89f8-160e1a62d27b" path="/var/lib/kubelet/pods/980e53f3-7f8f-4884-89f8-160e1a62d27b/volumes"
Jul 28 01:39:42.018183 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:42.018144    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"]
Jul 28 01:39:42.019906 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:42.019881    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0266d-qn5pp"]
Jul 28 01:39:42.946095 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:42.946062    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9dcc316-6e22-443b-abe6-5b1b7922c582" path="/var/lib/kubelet/pods/c9dcc316-6e22-443b-abe6-5b1b7922c582/volumes"
Jul 28 01:39:50.709439 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:39:50.709400    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:40:31.018787 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:31.018751    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"]
Jul 28 01:40:31.019363 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:31.019040    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy" containerID="cri-o://65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb" gracePeriod=30
Jul 28 01:40:36.285476 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.285407    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:40:36.325844 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.325808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-token\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.325910    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-envoy\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.325954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-podinfo\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.325982    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-socket\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-data\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326280 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9jwf\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-kube-api-access-l9jwf\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326280 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68750271-f3fe-4691-8422-ca51f2b2a93a-istiod-ca-cert\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326280 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326262    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-credential-socket\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326434 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326286    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-certs\") pod \"68750271-f3fe-4691-8422-ca51f2b2a93a\" (UID: \"68750271-f3fe-4691-8422-ca51f2b2a93a\") "
Jul 28 01:40:36.326434 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326295    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.326637 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326610    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68750271-f3fe-4691-8422-ca51f2b2a93a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:36.326704 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326631    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.326888 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.326865    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-data" (OuterVolumeSpecName: "istio-data") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.327111 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.327085    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.327187 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.327157    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.328300 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.328264    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.328524 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.328497    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:40:36.328657 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.328634    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-token" (OuterVolumeSpecName: "istio-token") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:36.328885 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.328862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-kube-api-access-l9jwf" (OuterVolumeSpecName: "kube-api-access-l9jwf") pod "68750271-f3fe-4691-8422-ca51f2b2a93a" (UID: "68750271-f3fe-4691-8422-ca51f2b2a93a"). InnerVolumeSpecName "kube-api-access-l9jwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:36.428135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428093    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428124    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428135    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428145    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428411 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428152    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428411 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428160    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68750271-f3fe-4691-8422-ca51f2b2a93a-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428411 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428168    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jwf\" (UniqueName: \"kubernetes.io/projected/68750271-f3fe-4691-8422-ca51f2b2a93a-kube-api-access-l9jwf\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.428411 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:36.428177    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68750271-f3fe-4691-8422-ca51f2b2a93a-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:37.083954 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.083906    2564 generic.go:356] "Generic (PLEG): container finished" podID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerID="65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb" exitCode=0
Jul 28 01:40:37.084163 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.083965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" event={"ID":"68750271-f3fe-4691-8422-ca51f2b2a93a","Type":"ContainerDied","Data":"65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb"}
Jul 28 01:40:37.084163 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.084041    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s" event={"ID":"68750271-f3fe-4691-8422-ca51f2b2a93a","Type":"ContainerDied","Data":"9a5e08212527ab04aac109dff70074918f9fe6ea5f60afcfb1dcd4dcbef6bdcc"}
Jul 28 01:40:37.084163 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.083984    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"
Jul 28 01:40:37.084163 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.084072    2564 scope.go:117] "RemoveContainer" containerID="65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb"
Jul 28 01:40:37.093522 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.093497    2564 scope.go:117] "RemoveContainer" containerID="2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"
Jul 28 01:40:37.102350 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.102328    2564 scope.go:117] "RemoveContainer" containerID="65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb"
Jul 28 01:40:37.102674 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:40:37.102653    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb\": container with ID starting with 65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb not found: ID does not exist" containerID="65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb"
Jul 28 01:40:37.102770 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.102685    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb"} err="failed to get container status \"65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb\": rpc error: code = NotFound desc = could not find container \"65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb\": container with ID starting with 65f85c9671eae5c45363b66b5c96929b5400fccc307ce72257c7fc05a7f520cb not found: ID does not exist"
Jul 28 01:40:37.102770 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.102704    2564 scope.go:117] "RemoveContainer" containerID="2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"
Jul 28 01:40:37.102770 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.102717    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"]
Jul 28 01:40:37.102991 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:40:37.102973    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da\": container with ID starting with 2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da not found: ID does not exist" containerID="2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"
Jul 28 01:40:37.103040 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.102998    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da"} err="failed to get container status \"2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da\": rpc error: code = NotFound desc = could not find container \"2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da\": container with ID starting with 2259b73d9aea1d8c356c6f69657c1b69f96c4782e4222fdfa4d984fcdfd651da not found: ID does not exist"
Jul 28 01:40:37.106532 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:37.106511    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0266d-openshift-default-85ddbbf84d-hpp6s"]
Jul 28 01:40:38.947549 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:38.947513    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" path="/var/lib/kubelet/pods/68750271-f3fe-4691-8422-ca51f2b2a93a/volumes"
Jul 28 01:40:42.528630 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.528580    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"]
Jul 28 01:40:42.529164 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.528921    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" podUID="cac2a642-5a1a-4b88-b989-e6a25350cec7" containerName="maas-api" containerID="cri-o://82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3" gracePeriod=30
Jul 28 01:40:42.790366 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.790306    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:40:42.888803 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.888772    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac2a642-5a1a-4b88-b989-e6a25350cec7-openshift-service-ca\") pod \"cac2a642-5a1a-4b88-b989-e6a25350cec7\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") "
Jul 28 01:40:42.888982 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.888814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdmqq\" (UniqueName: \"kubernetes.io/projected/cac2a642-5a1a-4b88-b989-e6a25350cec7-kube-api-access-jdmqq\") pod \"cac2a642-5a1a-4b88-b989-e6a25350cec7\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") "
Jul 28 01:40:42.888982 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.888948    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls\") pod \"cac2a642-5a1a-4b88-b989-e6a25350cec7\" (UID: \"cac2a642-5a1a-4b88-b989-e6a25350cec7\") "
Jul 28 01:40:42.889222 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.889199    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac2a642-5a1a-4b88-b989-e6a25350cec7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cac2a642-5a1a-4b88-b989-e6a25350cec7" (UID: "cac2a642-5a1a-4b88-b989-e6a25350cec7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:42.891020 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.890988    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac2a642-5a1a-4b88-b989-e6a25350cec7-kube-api-access-jdmqq" (OuterVolumeSpecName: "kube-api-access-jdmqq") pod "cac2a642-5a1a-4b88-b989-e6a25350cec7" (UID: "cac2a642-5a1a-4b88-b989-e6a25350cec7"). InnerVolumeSpecName "kube-api-access-jdmqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:42.891163 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.891071    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cac2a642-5a1a-4b88-b989-e6a25350cec7" (UID: "cac2a642-5a1a-4b88-b989-e6a25350cec7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:40:42.991085 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.990807    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac2a642-5a1a-4b88-b989-e6a25350cec7-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:42.991085 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.990898    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac2a642-5a1a-4b88-b989-e6a25350cec7-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:42.991085 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:42.990962    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmqq\" (UniqueName: \"kubernetes.io/projected/cac2a642-5a1a-4b88-b989-e6a25350cec7-kube-api-access-jdmqq\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:40:43.115059 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.114926    2564 generic.go:356] "Generic (PLEG): container finished" podID="cac2a642-5a1a-4b88-b989-e6a25350cec7" containerID="82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3" exitCode=0
Jul 28 01:40:43.115059 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.114997    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"
Jul 28 01:40:43.115059 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.115012    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" event={"ID":"cac2a642-5a1a-4b88-b989-e6a25350cec7","Type":"ContainerDied","Data":"82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3"}
Jul 28 01:40:43.115351 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.115077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v" event={"ID":"cac2a642-5a1a-4b88-b989-e6a25350cec7","Type":"ContainerDied","Data":"3dbc01ee4a6b743739eb65dfd6f63ec144b5618c8f94e59c85e3c5601b617073"}
Jul 28 01:40:43.115351 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.115109    2564 scope.go:117] "RemoveContainer" containerID="82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3"
Jul 28 01:40:43.124144 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.124126    2564 scope.go:117] "RemoveContainer" containerID="82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3"
Jul 28 01:40:43.124379 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:40:43.124360    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3\": container with ID starting with 82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3 not found: ID does not exist" containerID="82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3"
Jul 28 01:40:43.124428 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.124393    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3"} err="failed to get container status \"82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3\": rpc error: code = NotFound desc = could not find container \"82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3\": container with ID starting with 82fb9884e30d594050f898fe90a6334bd2bcb8e8b7acd66de5d1bffa74ec53c3 not found: ID does not exist"
Jul 28 01:40:43.130268 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.130245    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"]
Jul 28 01:40:43.136049 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:43.136025    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1ce59a-5d99cd49b6-l275v"]
Jul 28 01:40:44.945190 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:44.945159    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac2a642-5a1a-4b88-b989-e6a25350cec7" path="/var/lib/kubelet/pods/cac2a642-5a1a-4b88-b989-e6a25350cec7/volumes"
Jul 28 01:40:45.493187 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:40:45.493149    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:41:03.721503 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:03.721469    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"]
Jul 28 01:41:03.722276 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:03.722225    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" podUID="5a5951a4-d442-48a4-ab83-bdd529c9719a" containerName="maas-api" containerID="cri-o://1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62" gracePeriod=30
Jul 28 01:41:03.981792 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:03.981680    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:41:04.075504 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.075475    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hm9m\" (UniqueName: \"kubernetes.io/projected/5a5951a4-d442-48a4-ab83-bdd529c9719a-kube-api-access-7hm9m\") pod \"5a5951a4-d442-48a4-ab83-bdd529c9719a\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") "
Jul 28 01:41:04.075695 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.075534    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls\") pod \"5a5951a4-d442-48a4-ab83-bdd529c9719a\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") "
Jul 28 01:41:04.075695 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.075571    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5951a4-d442-48a4-ab83-bdd529c9719a-openshift-service-ca\") pod \"5a5951a4-d442-48a4-ab83-bdd529c9719a\" (UID: \"5a5951a4-d442-48a4-ab83-bdd529c9719a\") "
Jul 28 01:41:04.075965 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.075923    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5951a4-d442-48a4-ab83-bdd529c9719a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a5951a4-d442-48a4-ab83-bdd529c9719a" (UID: "5a5951a4-d442-48a4-ab83-bdd529c9719a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:04.077605 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.077580    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a5951a4-d442-48a4-ab83-bdd529c9719a" (UID: "5a5951a4-d442-48a4-ab83-bdd529c9719a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:04.077721 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.077622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5951a4-d442-48a4-ab83-bdd529c9719a-kube-api-access-7hm9m" (OuterVolumeSpecName: "kube-api-access-7hm9m") pod "5a5951a4-d442-48a4-ab83-bdd529c9719a" (UID: "5a5951a4-d442-48a4-ab83-bdd529c9719a"). InnerVolumeSpecName "kube-api-access-7hm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:04.177282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.177244    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hm9m\" (UniqueName: \"kubernetes.io/projected/5a5951a4-d442-48a4-ab83-bdd529c9719a-kube-api-access-7hm9m\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.177282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.177276    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a5951a4-d442-48a4-ab83-bdd529c9719a-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.177282 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.177286    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5951a4-d442-48a4-ab83-bdd529c9719a-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.221869 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.221840    2564 generic.go:356] "Generic (PLEG): container finished" podID="5a5951a4-d442-48a4-ab83-bdd529c9719a" containerID="1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62" exitCode=0
Jul 28 01:41:04.222064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.221881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" event={"ID":"5a5951a4-d442-48a4-ab83-bdd529c9719a","Type":"ContainerDied","Data":"1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62"}
Jul 28 01:41:04.222064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.221885    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"
Jul 28 01:41:04.222064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.221968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q" event={"ID":"5a5951a4-d442-48a4-ab83-bdd529c9719a","Type":"ContainerDied","Data":"bf1965e9e94dbb5d8d24ef8b3da30e1025585add5fd1390097a5c82bbfcc0553"}
Jul 28 01:41:04.222064 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.221991    2564 scope.go:117] "RemoveContainer" containerID="1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62"
Jul 28 01:41:04.230785 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.230765    2564 scope.go:117] "RemoveContainer" containerID="1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62"
Jul 28 01:41:04.231055 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:41:04.231035    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62\": container with ID starting with 1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62 not found: ID does not exist" containerID="1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62"
Jul 28 01:41:04.231114 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.231063    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62"} err="failed to get container status \"1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62\": rpc error: code = NotFound desc = could not find container \"1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62\": container with ID starting with 1b237f1cfdbd479e6e21957a65ade5fb3612b98c8757b23c07e05b8fbd4e6c62 not found: ID does not exist"
Jul 28 01:41:04.242823 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.242771    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"]
Jul 28 01:41:04.246224 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.246202    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14a1d0-57d6f5bd94-4xj5q"]
Jul 28 01:41:04.945046 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:04.945018    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5951a4-d442-48a4-ab83-bdd529c9719a" path="/var/lib/kubelet/pods/5a5951a4-d442-48a4-ab83-bdd529c9719a/volumes"
Jul 28 01:41:07.007760 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:07.007725    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:41:09.148399 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:09.148374    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:41:09.174689 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:09.174660    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:41:14.891527 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:14.891492    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"]
Jul 28 01:41:14.892000 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:14.891716    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy" containerID="cri-o://28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f" gracePeriod=30
Jul 28 01:41:15.266114 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266081    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"]
Jul 28 01:41:15.266473 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266460    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5951a4-d442-48a4-ab83-bdd529c9719a" containerName="maas-api"
Jul 28 01:41:15.266518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266474    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5951a4-d442-48a4-ab83-bdd529c9719a" containerName="maas-api"
Jul 28 01:41:15.266518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266486    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980e53f3-7f8f-4884-89f8-160e1a62d27b" containerName="maas-api"
Jul 28 01:41:15.266518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266491    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="980e53f3-7f8f-4884-89f8-160e1a62d27b" containerName="maas-api"
Jul 28 01:41:15.266518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266504    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy"
Jul 28 01:41:15.266518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266509    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy"
Jul 28 01:41:15.266518 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266516    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9dcc316-6e22-443b-abe6-5b1b7922c582" containerName="revoke-keys"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266522    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9dcc316-6e22-443b-abe6-5b1b7922c582" containerName="revoke-keys"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266531    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac2a642-5a1a-4b88-b989-e6a25350cec7" containerName="maas-api"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266536    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac2a642-5a1a-4b88-b989-e6a25350cec7" containerName="maas-api"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266546    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266551    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266608    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="980e53f3-7f8f-4884-89f8-160e1a62d27b" containerName="maas-api"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266617    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266623    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="68750271-f3fe-4691-8422-ca51f2b2a93a" containerName="istio-proxy"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266629    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac2a642-5a1a-4b88-b989-e6a25350cec7" containerName="maas-api"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266635    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9dcc316-6e22-443b-abe6-5b1b7922c582" containerName="revoke-keys"
Jul 28 01:41:15.266703 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.266641    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5951a4-d442-48a4-ab83-bdd529c9719a" containerName="maas-api"
Jul 28 01:41:15.271326 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.271304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:15.273834 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.273808    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:41:15.277229 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.277197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"]
Jul 28 01:41:15.372369 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.372333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3131e3-zndct\" (UID: \"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:15.473653 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.473614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3131e3-zndct\" (UID: \"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:15.474309 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.474288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3131e3-zndct\" (UID: \"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:15.584759 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.584675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:15.716593 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:15.716562    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"]
Jul 28 01:41:15.719357 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:41:15.719330    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f8a5f9_e94e_4f47_9bc7_887e0b3a7b73.slice/crio-a5ab08dce0c0cba811f496abaaa19f19e49b5d7f29cf27bc2dbeeff0d5f9e532 WatchSource:0}: Error finding container a5ab08dce0c0cba811f496abaaa19f19e49b5d7f29cf27bc2dbeeff0d5f9e532: Status 404 returned error can't find the container with id a5ab08dce0c0cba811f496abaaa19f19e49b5d7f29cf27bc2dbeeff0d5f9e532
Jul 28 01:41:16.282388 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:16.282357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct" event={"ID":"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73","Type":"ContainerStarted","Data":"d2a5dfbb939f3c96dd971a9752a9046b955db6cce82563ad4983c076f043415c"}
Jul 28 01:41:16.282868 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:16.282412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct" event={"ID":"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73","Type":"ContainerStarted","Data":"a5ab08dce0c0cba811f496abaaa19f19e49b5d7f29cf27bc2dbeeff0d5f9e532"}
Jul 28 01:41:16.299560 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:16.299510    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct" podStartSLOduration=1.299494201 podStartE2EDuration="1.299494201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:16.298038022 +0000 UTC m=+3307.853541195" watchObservedRunningTime="2026-07-28 01:41:16.299494201 +0000 UTC m=+3307.854997400"
Jul 28 01:41:17.288889 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:17.288853    2564 generic.go:356] "Generic (PLEG): container finished" podID="40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73" containerID="d2a5dfbb939f3c96dd971a9752a9046b955db6cce82563ad4983c076f043415c" exitCode=0
Jul 28 01:41:17.289310 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:17.288953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct" event={"ID":"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73","Type":"ContainerDied","Data":"d2a5dfbb939f3c96dd971a9752a9046b955db6cce82563ad4983c076f043415c"}
Jul 28 01:41:18.428525 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:18.428500    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:18.496379 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:18.496344    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73-maas-api-service-ca\") pod \"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73\" (UID: \"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73\") "
Jul 28 01:41:18.496693 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:18.496669    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73" (UID: "40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:18.597412 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:18.597329    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73-maas-api-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:19.299261 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:19.299183    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"
Jul 28 01:41:19.299261 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:19.299198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct" event={"ID":"40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73","Type":"ContainerDied","Data":"a5ab08dce0c0cba811f496abaaa19f19e49b5d7f29cf27bc2dbeeff0d5f9e532"}
Jul 28 01:41:19.299261 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:19.299238    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ab08dce0c0cba811f496abaaa19f19e49b5d7f29cf27bc2dbeeff0d5f9e532"
Jul 28 01:41:20.150681 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.150655    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:41:20.209826 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.209797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ghnv\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-kube-api-access-5ghnv\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.209832    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-podinfo\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.209855    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-envoy\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.209881    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-credential-socket\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.209919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-certs\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.209950    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-token\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-data\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210042 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210029    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istiod-ca-cert\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210436 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210059    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-socket\") pod \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\" (UID: \"358d73d2-54f3-4cb5-a13f-5c19dfd13142\") "
Jul 28 01:41:20.210436 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210250    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:20.210436 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210396    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-credential-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.210436 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210394    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:20.210648 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210471    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:20.210749 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.210725    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:20.211256 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.211222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-data" (OuterVolumeSpecName: "istio-data") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:20.212180 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.212154    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-kube-api-access-5ghnv" (OuterVolumeSpecName: "kube-api-access-5ghnv") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "kube-api-access-5ghnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:20.212263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.212176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-token" (OuterVolumeSpecName: "istio-token") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:20.212263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.212192    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:41:20.212263 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.212232    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "358d73d2-54f3-4cb5-a13f-5c19dfd13142" (UID: "358d73d2-54f3-4cb5-a13f-5c19dfd13142"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:20.305545 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.305516    2564 generic.go:356] "Generic (PLEG): container finished" podID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerID="28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f" exitCode=0
Jul 28 01:41:20.305743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.305558    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"
Jul 28 01:41:20.305743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.305596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" event={"ID":"358d73d2-54f3-4cb5-a13f-5c19dfd13142","Type":"ContainerDied","Data":"28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f"}
Jul 28 01:41:20.305743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.305695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4" event={"ID":"358d73d2-54f3-4cb5-a13f-5c19dfd13142","Type":"ContainerDied","Data":"feac699e8700e18127fa2b5c66fe61a9c413d96ec1460f6effeb27e773998a47"}
Jul 28 01:41:20.305743 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.305734    2564 scope.go:117] "RemoveContainer" containerID="28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f"
Jul 28 01:41:20.310866 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310846    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-data\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.310866 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310866    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istiod-ca-cert\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.311052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310876    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-socket\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.311052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310886    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ghnv\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-kube-api-access-5ghnv\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.311052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310895    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-podinfo\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.311052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310903    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-envoy\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.311052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310911    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358d73d2-54f3-4cb5-a13f-5c19dfd13142-workload-certs\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.311052 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.310921    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358d73d2-54f3-4cb5-a13f-5c19dfd13142-istio-token\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:20.314892 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.314871    2564 scope.go:117] "RemoveContainer" containerID="f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"
Jul 28 01:41:20.323493 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.323475    2564 scope.go:117] "RemoveContainer" containerID="28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f"
Jul 28 01:41:20.323725 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:41:20.323704    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f\": container with ID starting with 28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f not found: ID does not exist" containerID="28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f"
Jul 28 01:41:20.323835 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.323731    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f"} err="failed to get container status \"28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f\": rpc error: code = NotFound desc = could not find container \"28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f\": container with ID starting with 28144c8f0a0780ffe3f79bde4f0a77733d5b342c6aedbd5a3c7a2ee4f8a4b39f not found: ID does not exist"
Jul 28 01:41:20.323835 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.323748    2564 scope.go:117] "RemoveContainer" containerID="f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"
Jul 28 01:41:20.323985 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:41:20.323970    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92\": container with ID starting with f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92 not found: ID does not exist" containerID="f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"
Jul 28 01:41:20.324031 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.323995    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92"} err="failed to get container status \"f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92\": rpc error: code = NotFound desc = could not find container \"f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92\": container with ID starting with f17fda77dd9f0162a31a7374ebd6e0ba9302379889d5a8ca742dd0a6eb194e92 not found: ID does not exist"
Jul 28 01:41:20.326928 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.326908    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"]
Jul 28 01:41:20.330816 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.330797    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-14a1d0-openshift-default-7b5b5cbf9b-5r2j4"]
Jul 28 01:41:20.946910 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:20.946877    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" path="/var/lib/kubelet/pods/358d73d2-54f3-4cb5-a13f-5c19dfd13142/volumes"
Jul 28 01:41:25.648490 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.648455    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"]
Jul 28 01:41:25.648948 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.648653    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh" podUID="cdcf1949-f79b-4002-88a6-034f00791a64" containerName="maas-api" containerID="cri-o://c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418" gracePeriod=30
Jul 28 01:41:25.900324 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.900256    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:41:25.961156 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.961120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9r7x\" (UniqueName: \"kubernetes.io/projected/cdcf1949-f79b-4002-88a6-034f00791a64-kube-api-access-j9r7x\") pod \"cdcf1949-f79b-4002-88a6-034f00791a64\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") "
Jul 28 01:41:25.961340 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.961216    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcf1949-f79b-4002-88a6-034f00791a64-maas-api-tls\") pod \"cdcf1949-f79b-4002-88a6-034f00791a64\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") "
Jul 28 01:41:25.961340 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.961255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf1949-f79b-4002-88a6-034f00791a64-openshift-service-ca\") pod \"cdcf1949-f79b-4002-88a6-034f00791a64\" (UID: \"cdcf1949-f79b-4002-88a6-034f00791a64\") "
Jul 28 01:41:25.961921 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.961667    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcf1949-f79b-4002-88a6-034f00791a64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cdcf1949-f79b-4002-88a6-034f00791a64" (UID: "cdcf1949-f79b-4002-88a6-034f00791a64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:25.961921 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.961827    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf1949-f79b-4002-88a6-034f00791a64-openshift-service-ca\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:25.963459 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.963424    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcf1949-f79b-4002-88a6-034f00791a64-kube-api-access-j9r7x" (OuterVolumeSpecName: "kube-api-access-j9r7x") pod "cdcf1949-f79b-4002-88a6-034f00791a64" (UID: "cdcf1949-f79b-4002-88a6-034f00791a64"). InnerVolumeSpecName "kube-api-access-j9r7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:25.963735 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:25.963709    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcf1949-f79b-4002-88a6-034f00791a64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cdcf1949-f79b-4002-88a6-034f00791a64" (UID: "cdcf1949-f79b-4002-88a6-034f00791a64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:26.062269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.062230    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9r7x\" (UniqueName: \"kubernetes.io/projected/cdcf1949-f79b-4002-88a6-034f00791a64-kube-api-access-j9r7x\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.062269 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.062263    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcf1949-f79b-4002-88a6-034f00791a64-maas-api-tls\") on node \"ip-10-0-131-111.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.337307 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.337274    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdcf1949-f79b-4002-88a6-034f00791a64" containerID="c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418" exitCode=0
Jul 28 01:41:26.337506 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.337328    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"
Jul 28 01:41:26.337506 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.337361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh" event={"ID":"cdcf1949-f79b-4002-88a6-034f00791a64","Type":"ContainerDied","Data":"c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418"}
Jul 28 01:41:26.337506 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.337416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh" event={"ID":"cdcf1949-f79b-4002-88a6-034f00791a64","Type":"ContainerDied","Data":"457f7de68f0803167484218dcb77d0427d73a9fbc62c925f2c5c2fb700a97b3b"}
Jul 28 01:41:26.337506 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.337438    2564 scope.go:117] "RemoveContainer" containerID="c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418"
Jul 28 01:41:26.346777 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.346759    2564 scope.go:117] "RemoveContainer" containerID="c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418"
Jul 28 01:41:26.347075 ip-10-0-131-111 kubenswrapper[2564]: E0728 01:41:26.347056    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418\": container with ID starting with c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418 not found: ID does not exist" containerID="c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418"
Jul 28 01:41:26.347131 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.347084    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418"} err="failed to get container status \"c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418\": rpc error: code = NotFound desc = could not find container \"c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418\": container with ID starting with c812298916c5e80e3cd606254ae2afb57bf0dd4a6091ac8437f20f4129492418 not found: ID does not exist"
Jul 28 01:41:26.358979 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.358952    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"]
Jul 28 01:41:26.361996 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.361975    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3131e3-64f85b7498-lszsh"]
Jul 28 01:41:26.947579 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:26.947546    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdcf1949-f79b-4002-88a6-034f00791a64" path="/var/lib/kubelet/pods/cdcf1949-f79b-4002-88a6-034f00791a64/volumes"
Jul 28 01:41:28.699135 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:28.699097    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hrvn7"]
Jul 28 01:41:30.338169 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:30.338134    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"]
Jul 28 01:41:30.340789 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:30.340759    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3131e3-zndct"]
Jul 28 01:41:30.944562 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:30.944526    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73" path="/var/lib/kubelet/pods/40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73/volumes"
Jul 28 01:41:43.007386 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:43.007352    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hv8nt_85bf38c3-8c69-4fd7-be47-85555a13d6f5/manager/0.log"
Jul 28 01:41:43.480571 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:43.480546    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-2k65g_4ab83145-2bda-46ac-a23e-157afe8580c0/copy-manifests/0.log"
Jul 28 01:41:43.492715 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:43.492686    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-2k65g_4ab83145-2bda-46ac-a23e-157afe8580c0/manager/0.log"
Jul 28 01:41:44.227074 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:44.227044    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753340-mbzqf_3edcc857-1946-420c-bc26-e3f9258a635c/cleanup/0.log"
Jul 28 01:41:44.337009 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:44.336977    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753355-nfdz5_5d153bdc-5919-46ea-97db-1c8e78bf70f6/cleanup/0.log"
Jul 28 01:41:46.565975 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:46.565922    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hrvn7_8e5418a1-6f76-4259-a5c8-0ae65de93408/limitador/0.log"
Jul 28 01:41:46.684678 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:46.684647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-mv8gl_e003d4a9-9e4c-4c72-8e29-748e2865e74b/manager/0.log"
Jul 28 01:41:47.163445 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:47.163395    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cck8t_26d9efc1-24d6-4435-ad24-cb5a1712c834/discovery/0.log"
Jul 28 01:41:47.278763 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:47.278734    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-j295n_7bf5162b-e91f-4551-960b-a84ad77294a4/kube-auth-proxy/0.log"
Jul 28 01:41:47.866082 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:47.866041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d48b877f-xhtsx_ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5/router/0.log"
Jul 28 01:41:57.496342 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:57.496310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6vvw7_8c5c29c5-6324-44b2-ad00-5d8c4b9a36e8/global-pull-secret-syncer/0.log"
Jul 28 01:41:57.662011 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:57.661985    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qtd5t_580545c3-37c3-4853-89e3-3b35c757b895/konnectivity-agent/0.log"
Jul 28 01:41:57.715403 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:41:57.715373    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-111.ec2.internal_f04760098d23523919b1374e1df45b44/haproxy/0.log"
Jul 28 01:42:03.461318 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:03.461290    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hrvn7_8e5418a1-6f76-4259-a5c8-0ae65de93408/limitador/0.log"
Jul 28 01:42:03.537425 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:03.537391    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-mv8gl_e003d4a9-9e4c-4c72-8e29-748e2865e74b/manager/0.log"
Jul 28 01:42:05.320806 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:05.320776    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-9jrqd_05ad823c-b798-4bc0-95fe-2885ce46e53b/cluster-monitoring-operator/0.log"
Jul 28 01:42:05.486373 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:05.486339    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4v5w_e03dd829-a441-43bf-98bd-16395fd459ea/node-exporter/0.log"
Jul 28 01:42:05.507029 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:05.506998    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4v5w_e03dd829-a441-43bf-98bd-16395fd459ea/kube-rbac-proxy/0.log"
Jul 28 01:42:05.527010 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:05.526983    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4v5w_e03dd829-a441-43bf-98bd-16395fd459ea/init-textfile/0.log"
Jul 28 01:42:07.400473 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.400442    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-v5ct8_29f05ced-501b-4e37-b673-706d1a80b7af/networking-console-plugin/0.log"
Jul 28 01:42:07.594745 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.594714    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"]
Jul 28 01:42:07.595091 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595077    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy"
Jul 28 01:42:07.595142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595092    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy"
Jul 28 01:42:07.595142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595105    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy"
Jul 28 01:42:07.595142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595116    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy"
Jul 28 01:42:07.595142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595126    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73" containerName="revoke-keys"
Jul 28 01:42:07.595142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595131    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73" containerName="revoke-keys"
Jul 28 01:42:07.595142 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595141    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdcf1949-f79b-4002-88a6-034f00791a64" containerName="maas-api"
Jul 28 01:42:07.595331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595146    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdcf1949-f79b-4002-88a6-034f00791a64" containerName="maas-api"
Jul 28 01:42:07.595331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595215    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy"
Jul 28 01:42:07.595331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595224    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="358d73d2-54f3-4cb5-a13f-5c19dfd13142" containerName="istio-proxy"
Jul 28 01:42:07.595331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595233    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f8a5f9-e94e-4f47-9bc7-887e0b3a7b73" containerName="revoke-keys"
Jul 28 01:42:07.595331 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.595240    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdcf1949-f79b-4002-88a6-034f00791a64" containerName="maas-api"
Jul 28 01:42:07.598358 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.598337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.600857 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.600829    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2v9r2\"/\"default-dockercfg-d6gzj\""
Jul 28 01:42:07.601741 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.601723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2v9r2\"/\"kube-root-ca.crt\""
Jul 28 01:42:07.601845 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.601768    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2v9r2\"/\"openshift-service-ca.crt\""
Jul 28 01:42:07.608399 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.608367    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"]
Jul 28 01:42:07.726393 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.726354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-podres\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.726618 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.726407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmbs\" (UniqueName: \"kubernetes.io/projected/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-kube-api-access-nrmbs\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.726618 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.726510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-lib-modules\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.726618 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.726558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-proc\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.726769 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.726632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-sys\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.827805 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.827768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-sys\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.827818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-podres\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.827856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmbs\" (UniqueName: \"kubernetes.io/projected/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-kube-api-access-nrmbs\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.827888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-lib-modules\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.827912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-proc\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.827915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-sys\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828030 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.828022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-proc\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828288 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.828046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-podres\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.828288 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.828063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-lib-modules\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.836786 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.836762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmbs\" (UniqueName: \"kubernetes.io/projected/41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5-kube-api-access-nrmbs\") pod \"perf-node-gather-daemonset-slqpd\" (UID: \"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:07.908556 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:07.908519    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:08.042801 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.042758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"]
Jul 28 01:42:08.046114 ip-10-0-131-111 kubenswrapper[2564]: W0728 01:42:08.046073    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod41f4c7ad_42aa_4ae8_aa7d_884c04bd93a5.slice/crio-0a9e810229389a9868d82819bbfdb5c8082ba00497bb9f40af54462870652548 WatchSource:0}: Error finding container 0a9e810229389a9868d82819bbfdb5c8082ba00497bb9f40af54462870652548: Status 404 returned error can't find the container with id 0a9e810229389a9868d82819bbfdb5c8082ba00497bb9f40af54462870652548
Jul 28 01:42:08.068098 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.068051    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/1.log"
Jul 28 01:42:08.074132 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.074109    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-l64k4_b707cfdd-c2b5-4dc8-b386-b096fb2335c1/console-operator/2.log"
Jul 28 01:42:08.548751 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.548711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd" event={"ID":"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5","Type":"ContainerStarted","Data":"7941cf267f2f3022e2ed266281d0338d36444b3710fd2e368017dc096c458b48"}
Jul 28 01:42:08.549196 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.548882    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:08.549196 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.548906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd" event={"ID":"41f4c7ad-42aa-4ae8-aa7d-884c04bd93a5","Type":"ContainerStarted","Data":"0a9e810229389a9868d82819bbfdb5c8082ba00497bb9f40af54462870652548"}
Jul 28 01:42:08.565507 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.565452    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd" podStartSLOduration=1.565436687 podStartE2EDuration="1.565436687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:42:08.56459366 +0000 UTC m=+3360.120096821" watchObservedRunningTime="2026-07-28 01:42:08.565436687 +0000 UTC m=+3360.120939847"
Jul 28 01:42:08.629945 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:08.629912    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-88wn4_6b384be7-70d1-47a4-84be-fe3f3c8a2617/download-server/0.log"
Jul 28 01:42:09.218603 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:09.218568    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-z8cgh_36669965-8a4a-4c7b-89c0-3b1b5e995c3a/volume-data-source-validator/0.log"
Jul 28 01:42:10.352316 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:10.352290    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7vx6j_61ce6a7c-6678-4431-a4eb-6ceacba6faef/dns/0.log"
Jul 28 01:42:10.385538 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:10.385511    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7vx6j_61ce6a7c-6678-4431-a4eb-6ceacba6faef/kube-rbac-proxy/0.log"
Jul 28 01:42:10.455606 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:10.455582    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66whq_cfef6df0-7787-4f83-8768-06920413b08a/dns-node-resolver/0.log"
Jul 28 01:42:11.028280 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:11.028244    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d559cd4cf-d258t_ae4d208d-719d-4648-9422-0f394cc5463d/registry/0.log"
Jul 28 01:42:11.093545 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:11.093504    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gf5ph_31c2a323-206c-4f9c-b65c-555120aa0507/node-ca/0.log"
Jul 28 01:42:12.141275 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:12.141246    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cck8t_26d9efc1-24d6-4435-ad24-cb5a1712c834/discovery/0.log"
Jul 28 01:42:12.161449 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:12.161421    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-j295n_7bf5162b-e91f-4551-960b-a84ad77294a4/kube-auth-proxy/0.log"
Jul 28 01:42:12.329626 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:12.329598    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d48b877f-xhtsx_ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5/router/0.log"
Jul 28 01:42:13.136145 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:13.136080    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrvdd_493749e8-7d19-468f-a472-fceb6face574/serve-healthcheck-canary/0.log"
Jul 28 01:42:13.586804 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:13.586769    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-kkl92_6e6dacd7-8d50-4764-8313-e15b794713c3/insights-operator/0.log"
Jul 28 01:42:13.587354 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:13.587336    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-kkl92_6e6dacd7-8d50-4764-8313-e15b794713c3/insights-operator/1.log"
Jul 28 01:42:13.732926 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:13.732899    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5hrw_6bcdb825-e508-4e01-ad7b-65915bbd8b91/kube-rbac-proxy/0.log"
Jul 28 01:42:13.751914 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:13.751884    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5hrw_6bcdb825-e508-4e01-ad7b-65915bbd8b91/exporter/0.log"
Jul 28 01:42:13.771734 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:13.771701    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5hrw_6bcdb825-e508-4e01-ad7b-65915bbd8b91/extractor/0.log"
Jul 28 01:42:14.563953 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:14.563905    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-slqpd"
Jul 28 01:42:16.110726 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:16.110629    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hv8nt_85bf38c3-8c69-4fd7-be47-85555a13d6f5/manager/0.log"
Jul 28 01:42:16.455912 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:16.455874    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-2k65g_4ab83145-2bda-46ac-a23e-157afe8580c0/manager/0.log"
Jul 28 01:42:16.472339 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:16.472311    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-2k65g_4ab83145-2bda-46ac-a23e-157afe8580c0/copy-manifests/0.log"
Jul 28 01:42:17.910740 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:17.910710    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cck8t_26d9efc1-24d6-4435-ad24-cb5a1712c834/discovery/0.log"
Jul 28 01:42:17.920075 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:17.920052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-j295n_7bf5162b-e91f-4551-960b-a84ad77294a4/kube-auth-proxy/0.log"
Jul 28 01:42:18.047916 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:18.047828    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d48b877f-xhtsx_ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5/router/0.log"
Jul 28 01:42:19.634173 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:19.634089    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cck8t_26d9efc1-24d6-4435-ad24-cb5a1712c834/discovery/0.log"
Jul 28 01:42:19.646227 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:19.646203    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-j295n_7bf5162b-e91f-4551-960b-a84ad77294a4/kube-auth-proxy/0.log"
Jul 28 01:42:19.782120 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:19.782088    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d48b877f-xhtsx_ec93dc40-2060-4ffa-b9b8-a7f780d0a2b5/router/0.log"
Jul 28 01:42:22.818138 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:22.818111    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-ggxp4_a1b37d41-216e-418d-a163-e238c70acb9e/migrator/0.log"
Jul 28 01:42:22.836297 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:22.836274    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-ggxp4_a1b37d41-216e-418d-a163-e238c70acb9e/graceful-termination/0.log"
Jul 28 01:42:23.176097 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:23.176013    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xc692_91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4/kube-storage-version-migrator-operator/1.log"
Jul 28 01:42:23.176807 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:23.176791    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xc692_91a9d9d1-d089-4908-99bb-c0e7ea3e1ea4/kube-storage-version-migrator-operator/0.log"
Jul 28 01:42:24.495761 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.495725    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/kube-multus-additional-cni-plugins/0.log"
Jul 28 01:42:24.516758 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.516724    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/egress-router-binary-copy/0.log"
Jul 28 01:42:24.535615 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.535585    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/cni-plugins/0.log"
Jul 28 01:42:24.555464 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.555441    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/bond-cni-plugin/0.log"
Jul 28 01:42:24.575318 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.575295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/routeoverride-cni/0.log"
Jul 28 01:42:24.594521 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.594494    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/whereabouts-cni-bincopy/0.log"
Jul 28 01:42:24.614094 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.614073    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrzvt_a339eb81-8e58-4560-8c0b-d37f7750813e/whereabouts-cni/0.log"
Jul 28 01:42:24.651592 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.651565    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vwp85_9d258fb9-c885-41b7-88d1-93fe25e3c4eb/kube-multus/0.log"
Jul 28 01:42:24.716661 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.716636    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g88nz_0da18fd8-305e-4996-a531-2f569665d052/network-metrics-daemon/0.log"
Jul 28 01:42:24.732615 ip-10-0-131-111 kubenswrapper[2564]: I0728 01:42:24.732592    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g88nz_0da18fd8-305e-4996-a531-2f569665d052/kube-rbac-proxy/0.log"