Sep 10 13:28:52.145443 ip-10-0-130-202 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 13:28:52.145480 ip-10-0-130-202 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 13:28:52.145490 ip-10-0-130-202 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 13:28:52.145700 ip-10-0-130-202 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 13:29:03.244043 ip-10-0-130-202 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 13:29:03.244064 ip-10-0-130-202 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1e4b9a4597f54976ad5af2140b0a0fee --
Sep 10 13:30:55.853831 ip-10-0-130-202 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 13:30:56.267275 ip-10-0-130-202 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:30:56.267275 ip-10-0-130-202 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 13:30:56.267275 ip-10-0-130-202 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:30:56.267275 ip-10-0-130-202 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 13:30:56.267275 ip-10-0-130-202 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:30:56.267930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.267348    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 13:30:56.272842 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272822    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:30:56.272842 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272838    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:30:56.272842 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272842    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:30:56.272842 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272846    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272855    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272859    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272863    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272866    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272870    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272873    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272875    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272878    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272881    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272884    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272887    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272890    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272893    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272898    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272901    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272904    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272907    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272909    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272912    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:30:56.273004 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272915    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272917    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272920    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272923    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272925    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272928    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272933    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272937    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272940    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272942    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272946    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272950    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272955    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272958    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272962    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272966    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272969    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272972    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272978    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:30:56.273482 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272981    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272983    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272986    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272989    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272992    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272995    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.272999    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273001    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273004    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273007    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273010    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273016    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273019    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273022    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273024    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273028    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273031    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273033    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273036    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273039    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:30:56.273963 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273042    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273045    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273048    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273051    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273875    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273880    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273884    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273887    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273890    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273893    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273896    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273899    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273902    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273905    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273908    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273912    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273917    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273921    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273924    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273927    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:30:56.274451 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273930    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273933    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273938    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273941    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273944    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273947    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273949    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273952    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273958    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273961    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273964    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273967    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273969    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273972    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273975    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273978    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:30:56.274943 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.273981    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275053    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275060    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275063    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275066    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275071    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275074    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275077    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275080    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275083    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275086    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275088    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275093    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275097    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275100    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275103    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275106    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275109    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275114    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275117    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:30:56.275346 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275119    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275123    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275126    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275129    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275132    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275135    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275137    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275140    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275143    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275145    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275150    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275153    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275156    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275158    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275162    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275165    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275167    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275170    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275173    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275175    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:30:56.275927 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275178    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275180    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275184    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275189    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275191    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275196    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275199    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275210    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275212    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275215    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275218    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275221    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275224    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275226    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275231    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275234    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275237    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275240    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275244    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:30:56.276434 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275247    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275251    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275254    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275257    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275260    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275263    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275266    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275269    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275275    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275277    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275280    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275283    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275286    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275289    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275291    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275294    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275297    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275299    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275302    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275305    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:30:56.276925 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275307    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275313    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275315    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275319    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275322    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275324    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275327    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275330    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275332    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275335    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275337    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275340    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275345    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275348    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275351    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275353    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275356    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275359    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275362    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:30:56.277410 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275366    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275369    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275956    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275961    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275965    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275967    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275970    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275973    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275978    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275981    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275984    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275986    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275989    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275992    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275995    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.275998    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276001    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276003    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276006    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276009    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:30:56.277889 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276014    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276016    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276019    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276022    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276024    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276027    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276029    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276032    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276034    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276037    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276040    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276042    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276046    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276051    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276054    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276056    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276059    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276061    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276064    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276066    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:30:56.278371 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276069    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276072    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276074    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276077    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276080    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276082    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276087    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276090    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276093    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276096    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276098    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276101    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276103    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276107    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276109    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276112    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276114    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276117    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276122    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276124    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:30:56.278871 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276127    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276130    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276132    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276135    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276138    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276141    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276144    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276146    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276151    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276155    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276160    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276163    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276166    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276168    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276171    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276173    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276176    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276179    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276181    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276184    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:30:56.279342 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276187    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276190    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276192    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276197    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276200    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276204    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276208    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276210    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276215    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276220    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276223    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276226    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276229    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276234    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276237    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276240    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276246    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276248    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276251    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:30:56.279846 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276253    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.276256    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276521    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276573    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276598    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276604    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276610    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276615    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276630    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276636    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276640    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276645    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276649    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276654    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276659    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276664    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276669    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276676    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276681    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276687    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276695    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276702    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276709    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276714    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 13:30:56.280347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276722    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276730    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276734    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276742    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276747    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276756    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276763    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276769    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276776    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276783    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276791    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276796    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276803    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276810    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276817    2577 flags.go:64] FLAG: --enable-server="true"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276822    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276832    2577 flags.go:64] FLAG: --event-burst="100"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276839    2577 flags.go:64] FLAG: --event-qps="50"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276844    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276848    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276853    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276856    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276861    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276864    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276867    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 10 13:30:56.280965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276870    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276873    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276877    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276880    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276883    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276886    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276889    2577 flags.go:64] FLAG: --feature-gates=""
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276892    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276895    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276898    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276901    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276904    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276909    2577 flags.go:64] FLAG: --help="false"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276912    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276915    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276917    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276920    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276923    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276927    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276930    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276932    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276935    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276938    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 13:30:56.281548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276941    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276944    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276947    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276950    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276953    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276956    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276960    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276963    2577 flags.go:64] FLAG: --lock-file=""
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276966    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276969    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276971    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276976    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276980    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276983    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276985    2577 flags.go:64] FLAG: --logging-format="text"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276988    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276991    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276994    2577 flags.go:64] FLAG: --manifest-url=""
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.276997    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277001    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277004    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277009    2577 flags.go:64] FLAG: --max-pods="110"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277013    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277016    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277019    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 13:30:56.282147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277022    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277025    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277028    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277030    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277038    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277041    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277044    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277047    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277049    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277054    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277057    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277060    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277063    2577 flags.go:64] FLAG: --port="10250"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277066    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277070    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07b3f61ca2c53fe8e"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277073    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277076    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277079    2577 flags.go:64] FLAG: --register-node="true"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277082    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277086    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277089    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277092    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277094    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277098    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277101    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 13:30:56.282767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277104    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277106    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277109    2577 flags.go:64] FLAG: --runonce="false"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277113    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277117    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277119    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277122    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277125    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277128    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277131    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277134    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277137    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277139    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277142    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277145    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277148    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277151    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277153    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277159    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277162    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277164    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277171    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277174    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277176    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277179    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 13:30:56.283374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277182    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277185    2577 flags.go:64] FLAG: --v="2"
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277189    2577 flags.go:64] FLAG: --version="false"
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277193    2577 flags.go:64] FLAG: --vmodule=""
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277197    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.277200    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277305    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277309    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277312    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277314    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277319    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277322    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277324    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277327    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277330    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277332    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277335    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277338    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277341    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277343    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277346    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:30:56.284396 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277349    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277352    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277354    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277357    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277359    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277362    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277366    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277371    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277374    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277377    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277380    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277383    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277385    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277388    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277390    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277393    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277396    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277398    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277401    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:30:56.285310 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277404    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277406    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277410    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277414    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277418    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277421    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277424    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277427    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277430    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277433    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277435    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277438    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277440    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277443    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277446    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277449    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277451    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277454    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277457    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:30:56.286064 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277459    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277464    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277468    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277471    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277474    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277477    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277480    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277483    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277487    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277492    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277497    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277500    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277503    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277506    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277508    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277511    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277514    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277517    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277519    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:30:56.286528 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277522    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277524    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277527    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277530    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277532    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277535    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277537    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277540    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277542    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277545    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277547    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277550    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277553    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277555    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277559    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277562    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277564    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277567    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277569    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277572    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277589    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:30:56.287318 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277592    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277595    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277597    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277600    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277603    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.277606    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.278148    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.285760    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.285779    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.286616    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.286630    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286714    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286722    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286727    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286731    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:30:56.288179 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286735    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286739    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286744    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286748    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286752    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286757    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286761    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286765    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286769    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286774    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286778    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286782    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286786    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286791    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286795    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286799    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286803    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286808    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286812    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286816    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:30:56.288651 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286820    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286824    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286828    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286833    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286840    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286848    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286853    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286857    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286862    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286866    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286870    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286874    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286878    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286882    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286886    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286891    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286895    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286900    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286904    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:30:56.289516 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286908    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286912    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286916    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286920    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286924    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286931    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286937    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286943    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286947    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286951    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286955    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286960    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286964    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286969    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286973    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286977    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286981    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286985    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286990    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:30:56.290214 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.286995    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287000    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287004    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287008    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287012    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287016    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287020    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287024    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287028    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287032    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287036    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287040    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287045    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287049    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287053    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287057    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287061    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287065    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287069    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287073    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:30:56.290832 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287077    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287082    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287086    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287090    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287094    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287100    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287106    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287111    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287115    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287120    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287124    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287129    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287134    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287139    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287144    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287148    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:30:56.291315 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287153    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.287161    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287327    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287334    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287339    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287343    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287348    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287352    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287356    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287360    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287364    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287369    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287373    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287377    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287381    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:30:56.291792 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287385    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287389    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287393    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287396    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287401    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287405    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287409    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287413    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287417    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287422    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287426    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287430    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287434    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287438    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287442    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287448    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287452    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287456    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287460    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287464    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:30:56.292170 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287468    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287472    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287476    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287480    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287484    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287488    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287492    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287496    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287502    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287510    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287516    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287523    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287528    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287532    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287537    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287542    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287547    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287551    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287556    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:30:56.292763 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287560    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287564    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287568    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287572    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287594    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287599    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287603    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287608    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287612    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287617    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287622    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287626    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287630    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287635    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287639    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287643    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287648    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287652    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287656    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287660    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:30:56.293251 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287664    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287669    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287673    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287677    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287681    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287685    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287689    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287694    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287698    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287701    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287705    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287709    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287714    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287718    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287722    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287726    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287730    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287734    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287738    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287742    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:30:56.293829 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287746    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287750    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287754    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287758    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287764    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287770    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:56.287775    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.287782    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.287924    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.290451    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.291489    2577 server.go:1019] "Starting client certificate rotation"
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.291600    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:30:56.294674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.291628    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:30:56.295181 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.294405    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 13:30:56.295826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.295805    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 13:30:56.306178 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.306163    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 10 13:30:56.311401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.311386    2577 log.go:25] "Validated CRI v1 image API"
Sep 10 13:30:56.312384 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.312366    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 13:30:56.316428 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.316405    2577 fs.go:135] Filesystem UUIDs: map[6bbf7e65-cc1f-4ded-8829-4ae19aeb4894:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 88503d3b-18c3-4c8c-bde3-d2adc6fec963:/dev/nvme0n1p3]
Sep 10 13:30:56.316493 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.316427    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 13:30:56.323870 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.323762    2577 manager.go:216] Machine: {Timestamp:2026-09-10 13:30:56.32238229 +0000 UTC m=+0.347196983 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099326 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd54656c19bfbd18ef7b0854f84cc SystemUUID:ec2fd546-56c1-9bfb-d18e-f7b0854f84cc BootID:1e4b9a45-97f5-4976-ad5a-f2140b0a0fee Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:07:46:fb:99:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:07:46:fb:99:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:2d:1b:a0:a7:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 13:30:56.323976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.323878    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 13:30:56.324012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.323966    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 13:30:56.326881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.326856    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 13:30:56.327080 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.326904    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 13:30:56.327127 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.327111    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 13:30:56.327767 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.327756    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 13:30:56.327799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.327770    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 13:30:56.327799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.327777    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 13:30:56.328461 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.328451    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 13:30:56.329336 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.329325    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 13:30:56.332296 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332284    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 13:30:56.332415 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332405    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 13:30:56.332449 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332435    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 13:30:56.332449 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332447    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 13:30:56.332512 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332469    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 13:30:56.332512 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332488    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 10 13:30:56.332512 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332499    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 13:30:56.332864 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332853    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 13:30:56.332916 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.332872    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 13:30:56.335092 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.335075    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 13:30:56.337433 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.337417    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 13:30:56.337477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.337450    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 13:30:56.339281 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339269    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 13:30:56.339318 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339288    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 13:30:56.339318 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339294    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 13:30:56.339318 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339299    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 13:30:56.339318 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339308    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 13:30:56.339318 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339313    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 13:30:56.339456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339326    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 13:30:56.339456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339332    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 13:30:56.339456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339338    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 13:30:56.339456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339344    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 13:30:56.339456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339351    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 13:30:56.339456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.339369    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 13:30:56.340181 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.340171    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 13:30:56.340181 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.340181    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 13:30:56.343923 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.343911    2577 server.go:1268] "Started kubelet"
Sep 10 13:30:56.344022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.343972    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 13:30:56.344083 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.344027    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 13:30:56.344129 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.344095    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 13:30:56.344928 ip-10-0-130-202 systemd[1]: Started Kubernetes Kubelet.
Sep 10 13:30:56.345367 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.345347    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 13:30:56.346065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.346047    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:30:56.346147 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.346078    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:30:56.346215 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.346188    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:30:56.347481 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.347467    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 13:30:56.349945 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.349923    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 13:30:56.351044 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.350122    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-202.ec2.internal.18d3f8d1a7d5145c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-202.ec2.internal,UID:ip-10-0-130-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-202.ec2.internal,},FirstTimestamp:2026-09-10 13:30:56.343831644 +0000 UTC m=+0.368646337,LastTimestamp:2026-09-10 13:30:56.343831644 +0000 UTC m=+0.368646337,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-202.ec2.internal,}"
Sep 10 13:30:56.355191 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355173    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 13:30:56.355266 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355194    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 13:30:56.355327 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.355286    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.355753 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355740    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 13:30:56.355828 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355743    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 13:30:56.355828 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355777    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 13:30:56.355935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355867    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 13:30:56.355935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.355883    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 13:30:56.356072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356043    2577 factory.go:55] Registering systemd factory
Sep 10 13:30:56.356072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356059    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 10 13:30:56.356256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356243    2577 factory.go:153] Registering CRI-O factory
Sep 10 13:30:56.356256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356258    2577 factory.go:223] Registration of the crio container factory successfully
Sep 10 13:30:56.356320 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356304    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 13:30:56.356320 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356319    2577 factory.go:103] Registering Raw factory
Sep 10 13:30:56.356381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356330    2577 manager.go:1196] Started watching for new ooms in manager
Sep 10 13:30:56.356776 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.356753    2577 manager.go:349] Starting recovery of all containers
Sep 10 13:30:56.357298 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.357279    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:30:56.357375 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.357354    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 13:30:56.367829 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.367727    2577 manager.go:354] Recovery completed
Sep 10 13:30:56.369498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.369477    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 13:30:56.372604 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.372587    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.374362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.374342    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.374437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.374374    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.374437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.374385    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.375172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.375156    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 13:30:56.375172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.375167    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.375287 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.375169    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 13:30:56.375287 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.375282    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 13:30:56.376041 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.376024    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.376119 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.376055    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.376119 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.376079    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.376690 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.376675    2577 policy_none.go:49] "None policy: Start"
Sep 10 13:30:56.376753 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.376697    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 13:30:56.376753 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.376713    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 13:30:56.377997 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.377985    2577 policy_none.go:47] "Start"
Sep 10 13:30:56.387473 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.387406    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-202.ec2.internal.18d3f8d1a9a6ed7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-202.ec2.internal,UID:ip-10-0-130-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-202.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-202.ec2.internal,},FirstTimestamp:2026-09-10 13:30:56.374361469 +0000 UTC m=+0.399176162,LastTimestamp:2026-09-10 13:30:56.374361469 +0000 UTC m=+0.399176162,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-202.ec2.internal,}"
Sep 10 13:30:56.390935 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.390850    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-202.ec2.internal.18d3f8d1a9a7337e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-202.ec2.internal,UID:ip-10-0-130-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-202.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-202.ec2.internal,},FirstTimestamp:2026-09-10 13:30:56.37437939 +0000 UTC m=+0.399194084,LastTimestamp:2026-09-10 13:30:56.37437939 +0000 UTC m=+0.399194084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-202.ec2.internal,}"
Sep 10 13:30:56.401038 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.400964    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-202.ec2.internal.18d3f8d1a9a75884  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-202.ec2.internal,UID:ip-10-0-130-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-202.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-202.ec2.internal,},FirstTimestamp:2026-09-10 13:30:56.374388868 +0000 UTC m=+0.399203562,LastTimestamp:2026-09-10 13:30:56.374388868 +0000 UTC m=+0.399203562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-202.ec2.internal,}"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407131    2577 manager.go:337] "Starting Device Plugin manager"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.407164    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407173    2577 server.go:87] "Starting device plugin registration server"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407429    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407438    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407538    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407627    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407635    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.407818    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6t6mq"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.408100    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.408151    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.414497    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-202.ec2.internal.18d3f8d1a9a6ed7d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-202.ec2.internal.18d3f8d1a9a6ed7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-202.ec2.internal,UID:ip-10-0-130-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-202.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-202.ec2.internal,},FirstTimestamp:2026-09-10 13:30:56.374361469 +0000 UTC m=+0.399176162,LastTimestamp:2026-09-10 13:30:56.376039978 +0000 UTC m=+0.400854673,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-202.ec2.internal,}"
Sep 10 13:30:56.420381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.417414    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6t6mq"
Sep 10 13:30:56.444092 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.444067    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 13:30:56.445240 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.445224    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 13:30:56.445316 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.445268    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 13:30:56.445316 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.445291    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 13:30:56.445420 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.445331    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 13:30:56.447625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.447494    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:30:56.507899 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.507879    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.508655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.508640    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.508717 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.508668    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.508717 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.508680    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.508717 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.508703    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.515013 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.514998    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.515081 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.515016    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-202.ec2.internal\": node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.542912 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.542850    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.545927 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.545911    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"]
Sep 10 13:30:56.546016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.545961    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.548116 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.548098    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.548201 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.548125    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.548201 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.548135    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.549344 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.549328    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.549477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.549456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.549533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.549486    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.550096 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.550080    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.550180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.550110    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.550180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.550123    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.551074 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.551050    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.551152 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.551084    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.551152 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.551098    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.552258 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.552244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.552317 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.552267    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:30:56.552935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.552919    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:30:56.552990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.552948    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:30:56.552990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.552958    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:30:56.556836 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.556820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb5a6d65385ac6133bd7865d2c10033e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-202.ec2.internal\" (UID: \"bb5a6d65385ac6133bd7865d2c10033e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.556918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.556840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7c2fd2a540929d7cf219205287b1574-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal\" (UID: \"c7c2fd2a540929d7cf219205287b1574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.556918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.556856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c2fd2a540929d7cf219205287b1574-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal\" (UID: \"c7c2fd2a540929d7cf219205287b1574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.580114 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.580096    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-202.ec2.internal\" not found" node="ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.584389 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.584375    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-202.ec2.internal\" not found" node="ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.643802 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.643783    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.657082 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.657063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb5a6d65385ac6133bd7865d2c10033e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-202.ec2.internal\" (UID: \"bb5a6d65385ac6133bd7865d2c10033e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.657151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.657118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb5a6d65385ac6133bd7865d2c10033e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-202.ec2.internal\" (UID: \"bb5a6d65385ac6133bd7865d2c10033e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.657214 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.657164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7c2fd2a540929d7cf219205287b1574-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal\" (UID: \"c7c2fd2a540929d7cf219205287b1574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.657214 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.657204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c2fd2a540929d7cf219205287b1574-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal\" (UID: \"c7c2fd2a540929d7cf219205287b1574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.657302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.657252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7c2fd2a540929d7cf219205287b1574-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal\" (UID: \"c7c2fd2a540929d7cf219205287b1574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.657302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.657253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c2fd2a540929d7cf219205287b1574-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal\" (UID: \"c7c2fd2a540929d7cf219205287b1574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.744134 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.744104    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.844736 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.844707    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:56.882206 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.882188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.888271 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:56.888257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:56.945342 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:56.945318    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.045901 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:57.045875    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.146368 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:57.146314    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.248346 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:57.248320    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.291738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.291713    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 13:30:57.292110 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.291856    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 13:30:57.315244 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.315223    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:30:57.348726 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:57.348698    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.408930 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:57.408875    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5a6d65385ac6133bd7865d2c10033e.slice/crio-db395856b147f5391352dbc70d8613d47f48cfb1dc41619c68e7b289b1205535 WatchSource:0}: Error finding container db395856b147f5391352dbc70d8613d47f48cfb1dc41619c68e7b289b1205535: Status 404 returned error can't find the container with id db395856b147f5391352dbc70d8613d47f48cfb1dc41619c68e7b289b1205535
Sep 10 13:30:57.414613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.414598    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:30:57.422294 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.422269    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:25:56 +0000 UTC" deadline="2028-06-05 16:57:34.996634396 +0000 UTC"
Sep 10 13:30:57.422294 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.422294    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15219h26m37.574343066s"
Sep 10 13:30:57.448295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.448236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal" event={"ID":"bb5a6d65385ac6133bd7865d2c10033e","Type":"ContainerStarted","Data":"db395856b147f5391352dbc70d8613d47f48cfb1dc41619c68e7b289b1205535"}
Sep 10 13:30:57.449374 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:57.449357    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.554641 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:57.554618    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-202.ec2.internal\" not found"
Sep 10 13:30:57.566397 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.566377    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:30:57.655986 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.655923    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:57.672430 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.672406    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 13:30:57.673559 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.673543    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal"
Sep 10 13:30:57.686664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.686644    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 13:30:57.802242 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:57.802220    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:30:58.184006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.183977    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:30:58.333951 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.333926    2577 apiserver.go:52] "Watching apiserver"
Sep 10 13:30:58.343558 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.343535    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 13:30:58.345836 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.345806    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wsz7q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal","openshift-multus/multus-additional-cni-plugins-fshkc","openshift-multus/multus-zx6wj","openshift-network-operator/iptables-alerter-dn8p6","openshift-ovn-kubernetes/ovnkube-node-7xjdx","kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh","openshift-cluster-node-tuning-operator/tuned-vfg4z","openshift-dns/node-resolver-hmmtd","openshift-image-registry/node-ca-bhlmv","openshift-multus/network-metrics-daemon-txvf6","openshift-network-diagnostics/network-check-target-2vrct","kube-system/global-pull-secret-syncer-695ls"]
Sep 10 13:30:58.347495 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.347476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.348909 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.348887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:30:58.349004 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.348974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:30:58.350448 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.350417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.351995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.351953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zr66c\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353121    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nv7wk\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353231    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353370    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.353689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.353567    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 13:30:58.354618 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.354569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.356364 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.356177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.357212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.356857    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 13:30:58.357212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.357019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dzw42\""
Sep 10 13:30:58.357212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.357039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 13:30:58.357962 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.357937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.358049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.358017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mbdcr\""
Sep 10 13:30:58.358701 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.358330    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.358701 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.358558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 13:30:58.358850 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.358797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.362042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.361531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.362042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.361679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 13:30:58.362042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.361870    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkgg9\""
Sep 10 13:30:58.362042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.361877    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.362042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.361948    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 13:30:58.362346 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.362262    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 13:30:58.362346 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.362282    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 13:30:58.362346 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.362295    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2vwk7\""
Sep 10 13:30:58.363096 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.363070    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.363191 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.363083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.363381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.363364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:58.363452 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.363428    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:30:58.365625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-netns\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.365625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-multus-certs\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.365625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-etc-kubernetes\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.365625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-var-lib-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-cni-multus\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ac0c258-101b-451d-9899-e216adf7b7cc-host-slash\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-systemd-units\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-systemd\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-ovn\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-node-log\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.365901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-cni-netd\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49c5t\" (UniqueName: \"kubernetes.io/projected/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-kube-api-access-49c5t\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-system-cni-dir\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.365980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-slash\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovn-node-metrics-cert\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4sl\" (UniqueName: \"kubernetes.io/projected/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-kube-api-access-8n4sl\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-daemon-config\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-cnibin\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgztt\" (UniqueName: \"kubernetes.io/projected/3ac0c258-101b-451d-9899-e216adf7b7cc-kube-api-access-qgztt\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-run-netns\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-env-overrides\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-tmp-dir\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-cnibin\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.366312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ac0c258-101b-451d-9899-e216adf7b7cc-iptables-alerter-script\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-log-socket\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovnkube-config\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovnkube-script-lib\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-hosts-file\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-socket-dir-parent\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-cni-bin\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-kubelet\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-etc-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxfg\" (UniqueName: \"kubernetes.io/projected/17599da6-8770-4bd6-bc12-08af913e352a-kube-api-access-xhxfg\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-hostroot\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8684\" (UniqueName: \"kubernetes.io/projected/f5bb371c-b8f6-4344-bb06-7fa030b675c8-kube-api-access-s8684\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-k8s-cni-cncf-io\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.367136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-kubelet\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.368016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-conf-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.368016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.368016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-cni-bin\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.368016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-system-cni-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.368016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-cni-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.368016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.366999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5bb371c-b8f6-4344-bb06-7fa030b675c8-cni-binary-copy\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.368531 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.368511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.369675 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.369448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 13:30:58.369675 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.369470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jjwnn\""
Sep 10 13:30:58.369675 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.369525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 13:30:58.370309 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.370280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.371424 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.371404    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 13:30:58.372194 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.372174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.372348 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.372238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:30:58.372348 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.372323    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.372457 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.372431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.372597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.372541    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 13:30:58.372707 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.372574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rzqq\""
Sep 10 13:30:58.372940 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.372921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rm9b4\""
Sep 10 13:30:58.373018 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.373005    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 13:30:58.373213 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.373126    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 13:30:58.383175 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.383129    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 13:30:58.419026 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:58.418904    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c2fd2a540929d7cf219205287b1574.slice/crio-b7bc265682230e2e59228021cfb640bf95be7fcaad881da14e2e1bd19db90832 WatchSource:0}: Error finding container b7bc265682230e2e59228021cfb640bf95be7fcaad881da14e2e1bd19db90832: Status 404 returned error can't find the container with id b7bc265682230e2e59228021cfb640bf95be7fcaad881da14e2e1bd19db90832
Sep 10 13:30:58.449059 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.449002    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r7rh4"
Sep 10 13:30:58.450738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.450706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal" event={"ID":"c7c2fd2a540929d7cf219205287b1574","Type":"ContainerStarted","Data":"b7bc265682230e2e59228021cfb640bf95be7fcaad881da14e2e1bd19db90832"}
Sep 10 13:30:58.457063 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.457041    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 13:30:58.466344 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.466324    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r7rh4"
Sep 10 13:30:58.467420 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a404e583-94f1-455b-b237-16adac92fe09-dbus\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.467529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:58.467529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.467529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-kubernetes\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.467529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-var-lib-kubelet\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a404e583-94f1-455b-b237-16adac92fe09-kubelet-config\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-cnibin\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-log-socket\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4780b69-677c-48f7-b972-e3a66b6025df-host\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-kubelet\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-run\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/decf2b41-8503-4cbb-8e99-445be7e7a55d-tmp\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.467831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxfg\" (UniqueName: \"kubernetes.io/projected/17599da6-8770-4bd6-bc12-08af913e352a-kube-api-access-xhxfg\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-hostroot\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklpz\" (UniqueName: \"kubernetes.io/projected/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-kube-api-access-zklpz\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4780b69-677c-48f7-b972-e3a66b6025df-serviceca\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-k8s-cni-cncf-io\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-kubelet\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.467999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-cni-bin\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-system-cni-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-kubelet\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-cni-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5bb371c-b8f6-4344-bb06-7fa030b675c8-cni-binary-copy\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-kubelet\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-cni-bin\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-hostroot\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-system-cni-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-log-socket\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-cni-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-netns\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-multus-certs\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-etc-kubernetes\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-cnibin\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.468768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-var-lib-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-cni-multus\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-etc-kubernetes\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ac0c258-101b-451d-9899-e216adf7b7cc-host-slash\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-var-lib-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-systemd-units\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.468970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-ovn\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-k8s-cni-cncf-io\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-multus-certs\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-node-log\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-cni-multus\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5bb371c-b8f6-4344-bb06-7fa030b675c8-cni-binary-copy\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-ovn\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-cni-netd\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-modprobe-d\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.469172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-node-log\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-env-overrides\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ac0c258-101b-451d-9899-e216adf7b7cc-host-slash\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-systemd-units\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-run-netns\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49c5t\" (UniqueName: \"kubernetes.io/projected/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-kube-api-access-49c5t\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-cni-netd\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-slash\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovn-node-metrics-cert\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4sl\" (UniqueName: \"kubernetes.io/projected/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-kube-api-access-8n4sl\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-systemd\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-sys\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-slash\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.469878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pq8\" (UniqueName: \"kubernetes.io/projected/decf2b41-8503-4cbb-8e99-445be7e7a55d-kube-api-access-j6pq8\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.469984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-env-overrides\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-daemon-config\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-cnibin\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgztt\" (UniqueName: \"kubernetes.io/projected/3ac0c258-101b-451d-9899-e216adf7b7cc-kube-api-access-qgztt\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-run-netns\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-lib-modules\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-host\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49da1e9-edb6-47e7-a9f0-25bc1d3f7112-konnectivity-ca\") pod \"konnectivity-agent-wsz7q\" (UID: \"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112\") " pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-tmp-dir\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.470426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ac0c258-101b-451d-9899-e216adf7b7cc-iptables-alerter-script\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovnkube-config\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovnkube-script-lib\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysctl-d\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2kp\" (UniqueName: \"kubernetes.io/projected/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-kube-api-access-lx2kp\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-daemon-config\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-hosts-file\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-hosts-file\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-socket-dir-parent\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-cni-bin\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-socket-dir-parent\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-etc-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-registration-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49da1e9-edb6-47e7-a9f0-25bc1d3f7112-agent-certs\") pod \"konnectivity-agent-wsz7q\" (UID: \"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112\") " pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8684\" (UniqueName: \"kubernetes.io/projected/f5bb371c-b8f6-4344-bb06-7fa030b675c8-kube-api-access-s8684\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-tuned\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.470964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-sys-fs\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.470984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-host-var-lib-cni-bin\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-etc-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-tmp-dir\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-run-netns\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-cnibin\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ac0c258-101b-451d-9899-e216adf7b7cc-iptables-alerter-script\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.471736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17599da6-8770-4bd6-bc12-08af913e352a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2r8\" (UniqueName: \"kubernetes.io/projected/a4780b69-677c-48f7-b972-e3a66b6025df-kube-api-access-gp2r8\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovnkube-config\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-conf-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5bb371c-b8f6-4344-bb06-7fa030b675c8-multus-conf-dir\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.471992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovnkube-script-lib\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-device-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-systemd\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-systemd\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysconfig\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-run-openvswitch\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysctl-conf\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-system-cni-dir\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-socket-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-system-cni-dir\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17599da6-8770-4bd6-bc12-08af913e352a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.472724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.472647    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 13:30:58.477237 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.477087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-ovn-node-metrics-cert\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.504035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.503739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgztt\" (UniqueName: \"kubernetes.io/projected/3ac0c258-101b-451d-9899-e216adf7b7cc-kube-api-access-qgztt\") pod \"iptables-alerter-dn8p6\" (UID: \"3ac0c258-101b-451d-9899-e216adf7b7cc\") " pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.504744 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.504698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxfg\" (UniqueName: \"kubernetes.io/projected/17599da6-8770-4bd6-bc12-08af913e352a-kube-api-access-xhxfg\") pod \"multus-additional-cni-plugins-fshkc\" (UID: \"17599da6-8770-4bd6-bc12-08af913e352a\") " pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.505326 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.505302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4sl\" (UniqueName: \"kubernetes.io/projected/9fe748bb-3e14-48ac-af5e-bf8fafecf1ab-kube-api-access-8n4sl\") pod \"ovnkube-node-7xjdx\" (UID: \"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.505906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.505882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49c5t\" (UniqueName: \"kubernetes.io/projected/fb7c86e2-0c7a-417b-9b6f-ea9a33f95795-kube-api-access-49c5t\") pod \"node-resolver-hmmtd\" (UID: \"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795\") " pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.506872 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.506125    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:30:58.506872 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.506151    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:30:58.506872 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.506165    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:30:58.506872 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.506237    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.006215299 +0000 UTC m=+3.031029997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:30:58.509131 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.509111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8684\" (UniqueName: \"kubernetes.io/projected/f5bb371c-b8f6-4344-bb06-7fa030b675c8-kube-api-access-s8684\") pod \"multus-zx6wj\" (UID: \"f5bb371c-b8f6-4344-bb06-7fa030b675c8\") " pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.572896 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.572871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-socket-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.572990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.572909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a404e583-94f1-455b-b237-16adac92fe09-dbus\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.572990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.572930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:58.572990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.572957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-kubernetes\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.572990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.572982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-var-lib-kubelet\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a404e583-94f1-455b-b237-16adac92fe09-kubelet-config\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4780b69-677c-48f7-b972-e3a66b6025df-host\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.573052    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-run\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-socket-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a404e583-94f1-455b-b237-16adac92fe09-dbus\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/decf2b41-8503-4cbb-8e99-445be7e7a55d-tmp\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-kubernetes\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.573118    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.073098124 +0000 UTC m=+3.097912826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4780b69-677c-48f7-b972-e3a66b6025df-host\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a404e583-94f1-455b-b237-16adac92fe09-kubelet-config\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklpz\" (UniqueName: \"kubernetes.io/projected/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-kube-api-access-zklpz\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.573219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4780b69-677c-48f7-b972-e3a66b6025df-serviceca\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-run\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-modprobe-d\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-var-lib-kubelet\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-systemd\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-systemd\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.573436    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-modprobe-d\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-sys\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:58.573495    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.073474332 +0000 UTC m=+3.098289030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pq8\" (UniqueName: \"kubernetes.io/projected/decf2b41-8503-4cbb-8e99-445be7e7a55d-kube-api-access-j6pq8\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-sys\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-lib-modules\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-host\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49da1e9-edb6-47e7-a9f0-25bc1d3f7112-konnectivity-ca\") pod \"konnectivity-agent-wsz7q\" (UID: \"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112\") " pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.574007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysctl-d\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2kp\" (UniqueName: \"kubernetes.io/projected/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-kube-api-access-lx2kp\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-host\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-registration-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-lib-modules\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4780b69-677c-48f7-b972-e3a66b6025df-serviceca\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49da1e9-edb6-47e7-a9f0-25bc1d3f7112-agent-certs\") pod \"konnectivity-agent-wsz7q\" (UID: \"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112\") " pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-tuned\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysctl-d\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-sys-fs\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2r8\" (UniqueName: \"kubernetes.io/projected/a4780b69-677c-48f7-b972-e3a66b6025df-kube-api-access-gp2r8\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-device-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-registration-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.573955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysconfig\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysctl-conf\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.574799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-device-dir\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.575568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysctl-conf\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.575568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-sysconfig\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.575568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-sys-fs\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.575568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.574303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49da1e9-edb6-47e7-a9f0-25bc1d3f7112-konnectivity-ca\") pod \"konnectivity-agent-wsz7q\" (UID: \"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112\") " pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.576332 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.576285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/decf2b41-8503-4cbb-8e99-445be7e7a55d-tmp\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.576470 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.576437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/decf2b41-8503-4cbb-8e99-445be7e7a55d-etc-tuned\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.576826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.576802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49da1e9-edb6-47e7-a9f0-25bc1d3f7112-agent-certs\") pod \"konnectivity-agent-wsz7q\" (UID: \"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112\") " pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.592886 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.592860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklpz\" (UniqueName: \"kubernetes.io/projected/bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2-kube-api-access-zklpz\") pod \"aws-ebs-csi-driver-node-spplh\" (UID: \"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:58.596415 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.596395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pq8\" (UniqueName: \"kubernetes.io/projected/decf2b41-8503-4cbb-8e99-445be7e7a55d-kube-api-access-j6pq8\") pod \"tuned-vfg4z\" (UID: \"decf2b41-8503-4cbb-8e99-445be7e7a55d\") " pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.597910 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.597887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2kp\" (UniqueName: \"kubernetes.io/projected/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-kube-api-access-lx2kp\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:58.598022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.598003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2r8\" (UniqueName: \"kubernetes.io/projected/a4780b69-677c-48f7-b972-e3a66b6025df-kube-api-access-gp2r8\") pod \"node-ca-bhlmv\" (UID: \"a4780b69-677c-48f7-b972-e3a66b6025df\") " pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.661213 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.661185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmmtd"
Sep 10 13:30:58.668551 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.668529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fshkc"
Sep 10 13:30:58.678768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.678747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zx6wj"
Sep 10 13:30:58.685912 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.685889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dn8p6"
Sep 10 13:30:58.694813 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.694794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:30:58.702723 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.702671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfg4z"
Sep 10 13:30:58.709283 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.709259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:30:58.725659 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.725636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bhlmv"
Sep 10 13:30:58.732627 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:58.732607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh"
Sep 10 13:30:59.077933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.077899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:30:59.077933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.077937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.077958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078078    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078093    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078121    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078134    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078144    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.078128377 +0000 UTC m=+4.102943075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078078    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:59.078190 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078190    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.07817124 +0000 UTC m=+4.102985942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:30:59.078655 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.078229    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.078215806 +0000 UTC m=+4.103030503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:59.156504 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.156380    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4780b69_677c_48f7_b972_e3a66b6025df.slice/crio-708b5aff43623d4de1c42fb22e153ddef7a5d14e9e8229c77daf15e07c32c928 WatchSource:0}: Error finding container 708b5aff43623d4de1c42fb22e153ddef7a5d14e9e8229c77daf15e07c32c928: Status 404 returned error can't find the container with id 708b5aff43623d4de1c42fb22e153ddef7a5d14e9e8229c77daf15e07c32c928
Sep 10 13:30:59.156869 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.156851    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac0c258_101b_451d_9899_e216adf7b7cc.slice/crio-4dd238b42ff1fed4cb7b50144bc4f2c3301c1e5acb857cd41ea46b300b77e16f WatchSource:0}: Error finding container 4dd238b42ff1fed4cb7b50144bc4f2c3301c1e5acb857cd41ea46b300b77e16f: Status 404 returned error can't find the container with id 4dd238b42ff1fed4cb7b50144bc4f2c3301c1e5acb857cd41ea46b300b77e16f
Sep 10 13:30:59.178359 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.178334    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca95aaa_fa2d_4a8b_81f2_ee0c868b81c2.slice/crio-970b0ed318bb50daa572baeedc789b37aac4acae66fbcd87fb12d203233580d4 WatchSource:0}: Error finding container 970b0ed318bb50daa572baeedc789b37aac4acae66fbcd87fb12d203233580d4: Status 404 returned error can't find the container with id 970b0ed318bb50daa572baeedc789b37aac4acae66fbcd87fb12d203233580d4
Sep 10 13:30:59.182279 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.182251    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7c86e2_0c7a_417b_9b6f_ea9a33f95795.slice/crio-7d6c5068b7e42b4c63dd78f847069ba3cf884e281129494edec8214086c08a4f WatchSource:0}: Error finding container 7d6c5068b7e42b4c63dd78f847069ba3cf884e281129494edec8214086c08a4f: Status 404 returned error can't find the container with id 7d6c5068b7e42b4c63dd78f847069ba3cf884e281129494edec8214086c08a4f
Sep 10 13:30:59.182819 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.182749    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecf2b41_8503_4cbb_8e99_445be7e7a55d.slice/crio-069a317d19c3fed560a7eb22c8f7ffb341250257dcef9525ccde9d88e478d33c WatchSource:0}: Error finding container 069a317d19c3fed560a7eb22c8f7ffb341250257dcef9525ccde9d88e478d33c: Status 404 returned error can't find the container with id 069a317d19c3fed560a7eb22c8f7ffb341250257dcef9525ccde9d88e478d33c
Sep 10 13:30:59.183744 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.183505    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe748bb_3e14_48ac_af5e_bf8fafecf1ab.slice/crio-661e439cbd3b02c29d81c30371d458f88fc3ad86c8637da95a1e623dc5845931 WatchSource:0}: Error finding container 661e439cbd3b02c29d81c30371d458f88fc3ad86c8637da95a1e623dc5845931: Status 404 returned error can't find the container with id 661e439cbd3b02c29d81c30371d458f88fc3ad86c8637da95a1e623dc5845931
Sep 10 13:30:59.185013 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.184990    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bb371c_b8f6_4344_bb06_7fa030b675c8.slice/crio-e8d51adb9f0fa102251b875ce4233cc043228e9245c5ff20c811824785c33b69 WatchSource:0}: Error finding container e8d51adb9f0fa102251b875ce4233cc043228e9245c5ff20c811824785c33b69: Status 404 returned error can't find the container with id e8d51adb9f0fa102251b875ce4233cc043228e9245c5ff20c811824785c33b69
Sep 10 13:30:59.186367 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:30:59.186335    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49da1e9_edb6_47e7_a9f0_25bc1d3f7112.slice/crio-9f00577b10abbaefeb1d94cc2e43a04ca89eae168eaae59b53f8fb3fa1f95411 WatchSource:0}: Error finding container 9f00577b10abbaefeb1d94cc2e43a04ca89eae168eaae59b53f8fb3fa1f95411: Status 404 returned error can't find the container with id 9f00577b10abbaefeb1d94cc2e43a04ca89eae168eaae59b53f8fb3fa1f95411
Sep 10 13:30:59.446385 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.446310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:30:59.446918 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:30:59.446433    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:30:59.453638 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.453611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmmtd" event={"ID":"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795","Type":"ContainerStarted","Data":"7d6c5068b7e42b4c63dd78f847069ba3cf884e281129494edec8214086c08a4f"}
Sep 10 13:30:59.454783 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.454759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh" event={"ID":"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2","Type":"ContainerStarted","Data":"970b0ed318bb50daa572baeedc789b37aac4acae66fbcd87fb12d203233580d4"}
Sep 10 13:30:59.455904 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.455884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dn8p6" event={"ID":"3ac0c258-101b-451d-9899-e216adf7b7cc","Type":"ContainerStarted","Data":"4dd238b42ff1fed4cb7b50144bc4f2c3301c1e5acb857cd41ea46b300b77e16f"}
Sep 10 13:30:59.456929 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.456907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bhlmv" event={"ID":"a4780b69-677c-48f7-b972-e3a66b6025df","Type":"ContainerStarted","Data":"708b5aff43623d4de1c42fb22e153ddef7a5d14e9e8229c77daf15e07c32c928"}
Sep 10 13:30:59.457998 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.457978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zx6wj" event={"ID":"f5bb371c-b8f6-4344-bb06-7fa030b675c8","Type":"ContainerStarted","Data":"e8d51adb9f0fa102251b875ce4233cc043228e9245c5ff20c811824785c33b69"}
Sep 10 13:30:59.459167 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.459150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"661e439cbd3b02c29d81c30371d458f88fc3ad86c8637da95a1e623dc5845931"}
Sep 10 13:30:59.460257 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.460237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerStarted","Data":"a3743fd49f06b328ffc588003f53b949edf3d9ac28fd059ec542c6b5672d704f"}
Sep 10 13:30:59.463808 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.463787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal" event={"ID":"bb5a6d65385ac6133bd7865d2c10033e","Type":"ContainerStarted","Data":"fc9e4604add7256f74dc4473ee91dd6466c7b38a0fc5a5a3a3faaa6e253b6e3c"}
Sep 10 13:30:59.465012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.464991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wsz7q" event={"ID":"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112","Type":"ContainerStarted","Data":"9f00577b10abbaefeb1d94cc2e43a04ca89eae168eaae59b53f8fb3fa1f95411"}
Sep 10 13:30:59.466156 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.466136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfg4z" event={"ID":"decf2b41-8503-4cbb-8e99-445be7e7a55d","Type":"ContainerStarted","Data":"069a317d19c3fed560a7eb22c8f7ffb341250257dcef9525ccde9d88e478d33c"}
Sep 10 13:30:59.468521 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.468485    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:25:58 +0000 UTC" deadline="2028-03-06 16:54:02.136996322 +0000 UTC"
Sep 10 13:30:59.468591 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.468522    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13035h23m2.668478326s"
Sep 10 13:30:59.497572 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:30:59.497522    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-202.ec2.internal" podStartSLOduration=2.497505217 podStartE2EDuration="2.497505217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:59.496822482 +0000 UTC m=+3.521637187" watchObservedRunningTime="2026-09-10 13:30:59.497505217 +0000 UTC m=+3.522319925"
Sep 10 13:31:00.085721 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.085686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:00.085883 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.085743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:00.085883 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.085782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:00.086006 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.085928    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:00.086006 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.085991    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.08597382 +0000 UTC m=+6.110788522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:00.086445 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.086425    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:31:00.086541 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.086451    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:31:00.086541 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.086451    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:00.086541 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.086464    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:00.086541 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.086510    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.086492969 +0000 UTC m=+6.111307670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:00.086541 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.086533    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.086522423 +0000 UTC m=+6.111337115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:00.448716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.448639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:00.449145 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.448770    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:00.449145 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.448860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:00.449145 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:00.448952    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:00.468855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.468813    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:25:58 +0000 UTC" deadline="2028-02-15 10:18:24.034897175 +0000 UTC"
Sep 10 13:31:00.468855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:00.468854    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12548h47m23.566048036s"
Sep 10 13:31:01.446183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:01.445596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:01.446183 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:01.445788    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:02.101640 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:02.101606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:02.101660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:02.101693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.101832    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.101894    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.101872782 +0000 UTC m=+10.126687486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.101905    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.101930    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.101946    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.101999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.101982776 +0000 UTC m=+10.126797471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.102064    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:02.102295 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.102099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.102087877 +0000 UTC m=+10.126902586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:02.447144 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:02.446588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:02.447144 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:02.446608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:02.447144 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.446735    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:02.447144 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:02.446860    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:03.446399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:03.445908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:03.446399 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:03.446033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:04.445745 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:04.445713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:04.445892 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:04.445845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:04.446422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:04.446265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:04.446422 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:04.446379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:05.446522 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:05.446476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:05.447047 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:05.446628    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:06.133310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:06.133273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:06.133471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:06.133327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:06.133471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:06.133388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:06.133471 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133450    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133512    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133525    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.133503782 +0000 UTC m=+18.158318482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133530    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133529    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133544    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133600    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.133569858 +0000 UTC m=+18.158384570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:06.133652 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.133620    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.133610697 +0000 UTC m=+18.158425382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:06.453672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:06.453197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:06.453672 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.453332    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:06.454134 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:06.453828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:06.454312 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:06.454281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:07.445500 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:07.445464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:07.445694 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:07.445601    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:08.446259 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:08.446224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:08.446811 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:08.446380    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:08.446811 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:08.446453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:08.446811 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:08.446590    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:09.446199 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:09.446127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:09.446339 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:09.446240    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:10.445798 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:10.445767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:10.445992 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:10.445910    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:10.445992 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:10.445977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:10.446102 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:10.446074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:11.445743 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:11.445713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:11.446167 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:11.445826    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:12.446503 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:12.446460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:12.446503 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:12.446460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:12.446971 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:12.446632    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:12.446971 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:12.446691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:13.445703 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:13.445676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:13.445914 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:13.445769    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:14.195545 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:14.195497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:14.195551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:14.195597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195694    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195727    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195741    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195697    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195700    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.19577837 +0000 UTC m=+34.220593055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.195879886 +0000 UTC m=+34.220694581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:14.196000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.195926    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.195916489 +0000 UTC m=+34.220731174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:14.446447 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:14.446368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:14.446625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:14.446368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:14.446625 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.446488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:14.446625 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:14.446589    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:15.445745 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:15.445703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:15.446221 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:15.445833    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:16.446599 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:16.446553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:16.447043 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:16.446677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:16.447043 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:16.446735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:16.447043 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:16.446843    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:17.446649 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.446428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:17.446970 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:17.446726    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:17.507090 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.507058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfg4z" event={"ID":"decf2b41-8503-4cbb-8e99-445be7e7a55d","Type":"ContainerStarted","Data":"d7e2acfb913643fdf667c6bac9021a0184d2907bba56944f41db146cf1d441c1"}
Sep 10 13:31:17.510648 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.510620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zx6wj" event={"ID":"f5bb371c-b8f6-4344-bb06-7fa030b675c8","Type":"ContainerStarted","Data":"1693e2b627ce96104d696486193c151d6b7028bee26291b98e498f4ba93f9724"}
Sep 10 13:31:17.516155 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.516129    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:31:17.517697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.517414    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fe748bb-3e14-48ac-af5e-bf8fafecf1ab" containerID="6d60672c4f8323a15906c8362286d52e17178c5cb6a3d4caa694e58aed389e04" exitCode=1
Sep 10 13:31:17.517697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.517463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"e22ed7b4acb637dfeb3baa4974b99d4ee9e164fe1e131b5dd0cc03f9733e132e"}
Sep 10 13:31:17.517697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.517532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"428dd996c952e19a1c2b062fd14c29ba8ff2f65688f2a03795f668894a47f6c1"}
Sep 10 13:31:17.517697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.517599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerDied","Data":"6d60672c4f8323a15906c8362286d52e17178c5cb6a3d4caa694e58aed389e04"}
Sep 10 13:31:17.517697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.517646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"5a43a414769d41bd594ba2b1001cc461510ccf9b57d78bb6d2ea35851c8b5b90"}
Sep 10 13:31:17.573882 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.573827    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zx6wj" podStartSLOduration=3.5031731600000002 podStartE2EDuration="21.573806065s" totalImagesPullingTime="18.070632905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:17.573762079 +0000 UTC m=+21.598576782" watchObservedRunningTime="2026-09-10 13:31:17.573806065 +0000 UTC m=+21.598620767"
Sep 10 13:31:17.574235 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:17.574206    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vfg4z" podStartSLOduration=3.667610501 podStartE2EDuration="21.574198703s" totalImagesPullingTime="17.906588202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:17.545877504 +0000 UTC m=+21.570692209" watchObservedRunningTime="2026-09-10 13:31:17.574198703 +0000 UTC m=+21.599013405"
Sep 10 13:31:18.445706 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.445680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:18.445884 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.445712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:18.445884 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:18.445814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:18.446000 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:18.445936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:18.498509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.498490    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 13:31:18.522069 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.522047    2577 generic.go:356] "Generic (PLEG): container finished" podID="c7c2fd2a540929d7cf219205287b1574" containerID="56e83be40c2d1b5003a5c53dc5acad267c514b587195e27422e92d2c29041263" exitCode=0
Sep 10 13:31:18.522163 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.522114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal" event={"ID":"c7c2fd2a540929d7cf219205287b1574","Type":"ContainerDied","Data":"56e83be40c2d1b5003a5c53dc5acad267c514b587195e27422e92d2c29041263"}
Sep 10 13:31:18.523592 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.523562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wsz7q" event={"ID":"b49da1e9-edb6-47e7-a9f0-25bc1d3f7112","Type":"ContainerStarted","Data":"356383c53164209bcda7f01950e5fb3f30478d8f9fe44d6bf1a2809d49013f30"}
Sep 10 13:31:18.525061 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.525042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmmtd" event={"ID":"fb7c86e2-0c7a-417b-9b6f-ea9a33f95795","Type":"ContainerStarted","Data":"39f0b19c08e58120e6747cb34f702ae535db0c11b48fb25042b1993d51c80214"}
Sep 10 13:31:18.529979 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.529949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh" event={"ID":"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2","Type":"ContainerStarted","Data":"1edd2a4f70b85e1ce833e47776fad26730cbd9b526e19eaacd7ae49d28718585"}
Sep 10 13:31:18.530071 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.530028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh" event={"ID":"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2","Type":"ContainerStarted","Data":"b50a3c48cf5c4581ac177f54d7e7e8812b171f0dfe9515c7d56ccbb557007348"}
Sep 10 13:31:18.531405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.531386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dn8p6" event={"ID":"3ac0c258-101b-451d-9899-e216adf7b7cc","Type":"ContainerStarted","Data":"9d4b3c30bb0ee103b56ed7b5ab869aea058e78c9819d15c29489be38dfad5f82"}
Sep 10 13:31:18.532867 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.532847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bhlmv" event={"ID":"a4780b69-677c-48f7-b972-e3a66b6025df","Type":"ContainerStarted","Data":"5c68e0a5fca6d314b1d7ae67c7aff7ea410482786bb1fdc53d37149d799cb90d"}
Sep 10 13:31:18.535550 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.535532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:31:18.535938 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.535918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"3502ecc1a764a8e12f4b05ed0916ff989f79863f4da7a74c43eca039abc7c706"}
Sep 10 13:31:18.536008 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.535978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"8f3365c5f82691445c16d59d8b4a99fdb29fe9971e027c0c652ced5bace8e58f"}
Sep 10 13:31:18.537335 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.537314    2577 generic.go:356] "Generic (PLEG): container finished" podID="17599da6-8770-4bd6-bc12-08af913e352a" containerID="085dd1ac299782559df3ec87074fb5cf577cc8cd4e7a1a34faf554308ad73c0f" exitCode=0
Sep 10 13:31:18.537417 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.537396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerDied","Data":"085dd1ac299782559df3ec87074fb5cf577cc8cd4e7a1a34faf554308ad73c0f"}
Sep 10 13:31:18.567265 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.567229    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wsz7q" podStartSLOduration=3.663072089 podStartE2EDuration="21.567216252s" totalImagesPullingTime="17.904144163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:18.566845488 +0000 UTC m=+22.591660186" watchObservedRunningTime="2026-09-10 13:31:18.567216252 +0000 UTC m=+22.592030955"
Sep 10 13:31:18.586930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.586899    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bhlmv" podStartSLOduration=3.697594884 podStartE2EDuration="21.586889505s" totalImagesPullingTime="17.889294621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:18.586785004 +0000 UTC m=+22.611599705" watchObservedRunningTime="2026-09-10 13:31:18.586889505 +0000 UTC m=+22.611704207"
Sep 10 13:31:18.612689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.612653    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmmtd" podStartSLOduration=12.263931578 podStartE2EDuration="22.612639299s" totalImagesPullingTime="10.348707721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:18.612543659 +0000 UTC m=+22.637358360" watchObservedRunningTime="2026-09-10 13:31:18.612639299 +0000 UTC m=+22.637454001"
Sep 10 13:31:18.648469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:18.648432    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dn8p6" podStartSLOduration=12.28782439 podStartE2EDuration="22.64842379s" totalImagesPullingTime="10.3605994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:18.648151604 +0000 UTC m=+22.672966296" watchObservedRunningTime="2026-09-10 13:31:18.64842379 +0000 UTC m=+22.673238491"
Sep 10 13:31:19.419968 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.419821    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:31:18.498505484Z","UUID":"161224f0-de81-43fc-816d-39a2b02282e5","Handler":null,"Name":"","Endpoint":""}
Sep 10 13:31:19.421501 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.421480    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 13:31:19.421626 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.421515    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 13:31:19.445969 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.445942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:19.446084 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:19.446060    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:19.541920 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.541683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal" event={"ID":"c7c2fd2a540929d7cf219205287b1574","Type":"ContainerStarted","Data":"1d71fbd98cf1c9ea35d8c9daa537c824a8c89ac7d09a2ccbc2b615ba0b59c4a3"}
Sep 10 13:31:19.544118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.544089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh" event={"ID":"bca95aaa-fa2d-4a8b-81f2-ee0c868b81c2","Type":"ContainerStarted","Data":"1e147e2927f2a4b1749c494f8267d9da70007a549f6740f7dd9930ac5555a315"}
Sep 10 13:31:19.578303 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.578256    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-202.ec2.internal" podStartSLOduration=22.578242506 podStartE2EDuration="22.578242506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:19.577872819 +0000 UTC m=+23.602687521" watchObservedRunningTime="2026-09-10 13:31:19.578242506 +0000 UTC m=+23.603057208"
Sep 10 13:31:19.601483 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:19.601444    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spplh" podStartSLOduration=3.042920505 podStartE2EDuration="22.601430499s" totalImagesPullingTime="19.558509994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:19.600972558 +0000 UTC m=+23.625787259" watchObservedRunningTime="2026-09-10 13:31:19.601430499 +0000 UTC m=+23.626245200"
Sep 10 13:31:20.446551 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:20.446521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:20.446821 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:20.446663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:20.446821 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:20.446701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:20.446821 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:20.446777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:20.550160 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:20.550135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:31:20.550635 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:20.550609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"16bf6f1dc0723785f094c0d543ce04b8c993f6f8bbaf58620c50e9898a29fb05"}
Sep 10 13:31:21.445800 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:21.445768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:21.445952 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:21.445866    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:22.449052 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:22.449007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:22.449425 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:22.449123    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:22.449425 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:22.449188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:22.449425 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:22.449299    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:23.316405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.316245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:31:23.316916 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.316897    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:31:23.445985 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.445926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:23.446057 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:23.446037    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:23.559709 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.559655    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:31:23.560080 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.560032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"e2da42a111d0fc64a0e968ecf5b514f32d830313ba53f4c33bf59a92662c7e12"}
Sep 10 13:31:23.560298 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.560261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:31:23.560472 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.560338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:31:23.560539 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.560495    2577 scope.go:117] "RemoveContainer" containerID="6d60672c4f8323a15906c8362286d52e17178c5cb6a3d4caa694e58aed389e04"
Sep 10 13:31:23.562079 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.562060    2577 generic.go:356] "Generic (PLEG): container finished" podID="17599da6-8770-4bd6-bc12-08af913e352a" containerID="d7d2a917297ac810b1c9a01d0cb2579ae59b505336913d87e4173a398ab8beeb" exitCode=0
Sep 10 13:31:23.562180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.562132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerDied","Data":"d7d2a917297ac810b1c9a01d0cb2579ae59b505336913d87e4173a398ab8beeb"}
Sep 10 13:31:23.562430 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.562388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:31:23.562777 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.562760    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wsz7q"
Sep 10 13:31:23.578718 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:23.578701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:31:24.445831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.445808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:24.445956 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.445842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:24.445956 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:24.445924    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:24.446065 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:24.446042    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:24.567221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.567196    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:31:24.567679 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.567572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" event={"ID":"9fe748bb-3e14-48ac-af5e-bf8fafecf1ab","Type":"ContainerStarted","Data":"a89f685012bcc2926345e0b9d39c0cd936014281399bc01bf6d4092af9e35e84"}
Sep 10 13:31:24.567807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.567788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:31:24.569880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.569859    2577 generic.go:356] "Generic (PLEG): container finished" podID="17599da6-8770-4bd6-bc12-08af913e352a" containerID="4c2d9f2fca0063408b920dae3620863dc09706fcfeb1c53936e45d8aa4f8618a" exitCode=0
Sep 10 13:31:24.569996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.569942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerDied","Data":"4c2d9f2fca0063408b920dae3620863dc09706fcfeb1c53936e45d8aa4f8618a"}
Sep 10 13:31:24.581741 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.581723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:31:24.609338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:24.607009    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx" podStartSLOduration=10.664319272 podStartE2EDuration="28.606983472s" totalImagesPullingTime="17.9426642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:24.606825104 +0000 UTC m=+28.631639801" watchObservedRunningTime="2026-09-10 13:31:24.606983472 +0000 UTC m=+28.631798174"
Sep 10 13:31:25.446469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:25.446435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:25.446567 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:25.446539    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:25.575095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:25.575037    2577 generic.go:356] "Generic (PLEG): container finished" podID="17599da6-8770-4bd6-bc12-08af913e352a" containerID="218c28e480e133a527ddd4ed562b0252856d070bab3588c5124f6f892dbeb728" exitCode=0
Sep 10 13:31:25.575429 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:25.575128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerDied","Data":"218c28e480e133a527ddd4ed562b0252856d070bab3588c5124f6f892dbeb728"}
Sep 10 13:31:26.446822 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:26.446787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:26.446995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:26.446957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:26.447103 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:26.447067    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:26.447103 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:26.446921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:27.446110 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:27.446083    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:27.446696 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:27.446194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:28.446429 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:28.446397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:28.446429 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:28.446410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:28.446980 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:28.446531    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:28.446980 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:28.446685    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:29.446054 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:29.446028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:29.446216 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:29.446128    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:30.218971 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:30.218938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:30.219006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:30.219042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219103    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219152    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret podName:a404e583-94f1-455b-b237-16adac92fe09 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:02.219157258 +0000 UTC m=+66.243971956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret") pod "global-pull-secret-syncer-695ls" (UID: "a404e583-94f1-455b-b237-16adac92fe09") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219203    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs podName:f81e5544-e11d-4c22-bbbc-e09f9d1034a9 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:02.219186495 +0000 UTC m=+66.244001198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs") pod "network-metrics-daemon-txvf6" (UID: "f81e5544-e11d-4c22-bbbc-e09f9d1034a9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219214    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219244    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219261    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5srzq for pod openshift-network-diagnostics/network-check-target-2vrct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:30.219498 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.219328    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq podName:40df8a05-6418-4830-8b6a-0d810371127a nodeName:}" failed. No retries permitted until 2026-09-10 13:32:02.219316639 +0000 UTC m=+66.244131325 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5srzq" (UniqueName: "kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq") pod "network-check-target-2vrct" (UID: "40df8a05-6418-4830-8b6a-0d810371127a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:30.446501 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:30.446452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:30.446708 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.446615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:30.446708 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:30.446671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:30.446834 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:30.446768    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:31.446439 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:31.446415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:31.446799 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:31.446499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:31.591438 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:31.591411    2577 generic.go:356] "Generic (PLEG): container finished" podID="17599da6-8770-4bd6-bc12-08af913e352a" containerID="491e7a6023a92930b921a3890d8f8ea6d31c553ee2f2ec3bd049937860d7e353" exitCode=0
Sep 10 13:31:31.591568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:31.591459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerDied","Data":"491e7a6023a92930b921a3890d8f8ea6d31c553ee2f2ec3bd049937860d7e353"}
Sep 10 13:31:32.445841 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:32.445816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:32.445998 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:32.445913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:32.446045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:32.445997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:32.446125 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:32.446104    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:32.596350 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:32.596319    2577 generic.go:356] "Generic (PLEG): container finished" podID="17599da6-8770-4bd6-bc12-08af913e352a" containerID="07b3f4e8eaad5ee7b35def30f75931a2d2649782cf2cf86048fc1ef8f9362532" exitCode=0
Sep 10 13:31:32.596350 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:32.596352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerDied","Data":"07b3f4e8eaad5ee7b35def30f75931a2d2649782cf2cf86048fc1ef8f9362532"}
Sep 10 13:31:33.445509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:33.445477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:33.445680 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:33.445613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:33.601543 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:33.601507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fshkc" event={"ID":"17599da6-8770-4bd6-bc12-08af913e352a","Type":"ContainerStarted","Data":"bdcc7cc4e694708258df3bfd9a622958f25d5d74eae6698633f25f7520aca32a"}
Sep 10 13:31:33.627719 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:33.627681    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fshkc" podStartSLOduration=8.174867999 podStartE2EDuration="37.627668448s" totalImagesPullingTime="29.452800449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:33.626524621 +0000 UTC m=+37.651339323" watchObservedRunningTime="2026-09-10 13:31:33.627668448 +0000 UTC m=+37.652483152"
Sep 10 13:31:34.446116 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:34.446084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:34.446258 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:34.446193    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:34.446258 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:34.446248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:34.446357 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:34.446339    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:35.445930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:35.445898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:35.446308 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:35.446001    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:36.103062 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:36.102838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-txvf6"]
Sep 10 13:31:36.103316 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:36.103190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:36.103399 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:36.103319    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:36.105523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:36.105501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-695ls"]
Sep 10 13:31:36.105637 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:36.105612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:36.105724 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:36.105706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:36.126869 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:36.126806    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vrct"]
Sep 10 13:31:36.126958 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:36.126881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:36.127001 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:36.126973    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:37.445734 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:37.445707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:37.446087 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:37.445710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:37.446087 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:37.445796    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:37.446087 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:37.445886    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:38.448819 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:38.448790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:38.449181 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:38.448888    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:39.446088 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:39.446060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:39.446088 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:39.446082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:39.446284 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:39.446167    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-695ls" podUID="a404e583-94f1-455b-b237-16adac92fe09"
Sep 10 13:31:39.446284 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:39.446272    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vrct" podUID="40df8a05-6418-4830-8b6a-0d810371127a"
Sep 10 13:31:40.445961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.445931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:40.446416 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:40.446039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txvf6" podUID="f81e5544-e11d-4c22-bbbc-e09f9d1034a9"
Sep 10 13:31:40.759058 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.759035    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-202.ec2.internal" event="NodeReady"
Sep 10 13:31:40.759147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.759137    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 13:31:40.810219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.810196    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-5dd89"]
Sep 10 13:31:40.824588 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.824557    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d7jkb"]
Sep 10 13:31:40.824693 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.824677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:40.827062 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.827041    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 13:31:40.827307 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.827288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 13:31:40.827425 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.827353    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 13:31:40.827490 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.827438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 13:31:40.827490 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.827460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 13:31:40.827620 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.827601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ps95x\""
Sep 10 13:31:40.832850 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.832365    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8zp5t"]
Sep 10 13:31:40.832850 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.832441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:40.836975 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.836952    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w88gf\""
Sep 10 13:31:40.838310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.837224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 13:31:40.838310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.837305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 13:31:40.838310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.837329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 13:31:40.844629 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.844612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d7jkb"]
Sep 10 13:31:40.844733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.844639    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-5dd89"]
Sep 10 13:31:40.844733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.844655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zp5t"]
Sep 10 13:31:40.844733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.844674    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p2xt2"]
Sep 10 13:31:40.844733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.844682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:40.847021 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.846997    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 13:31:40.847021 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.847020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 13:31:40.847169 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.847064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 13:31:40.847225 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.847209    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2d75h\""
Sep 10 13:31:40.847259 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.847246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 13:31:40.854805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.854790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:40.856888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.856872    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 13:31:40.856966 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.856918    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 13:31:40.857166 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.857154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9764\""
Sep 10 13:31:40.860281 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.860173    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2xt2"]
Sep 10 13:31:40.901338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb53d9a8-99c6-4a02-963e-0a5efa640280-config-volume\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:40.901338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3758b02-ff8d-4023-8e56-10f9d5e25be0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:40.901338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8gz\" (UniqueName: \"kubernetes.io/projected/a3758b02-ff8d-4023-8e56-10f9d5e25be0-kube-api-access-8g8gz\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8lw\" (UniqueName: \"kubernetes.io/projected/68a8e641-6318-47ab-9605-39cc5dfb3c81-kube-api-access-ch8lw\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb53d9a8-99c6-4a02-963e-0a5efa640280-metrics-tls\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7msr\" (UniqueName: \"kubernetes.io/projected/fb53d9a8-99c6-4a02-963e-0a5efa640280-kube-api-access-h7msr\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e27269-7607-4483-b8c3-b161b07da691-cert\") pod \"ingress-canary-d7jkb\" (UID: \"a8e27269-7607-4483-b8c3-b161b07da691\") " pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68a8e641-6318-47ab-9605-39cc5dfb3c81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:40.901525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwtg5\" (UniqueName: \"kubernetes.io/projected/a8e27269-7607-4483-b8c3-b161b07da691-kube-api-access-vwtg5\") pod \"ingress-canary-d7jkb\" (UID: \"a8e27269-7607-4483-b8c3-b161b07da691\") " pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:40.901802 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68a8e641-6318-47ab-9605-39cc5dfb3c81-crio-socket\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:40.901802 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a8e641-6318-47ab-9605-39cc5dfb3c81-data-volume\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:40.901802 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb53d9a8-99c6-4a02-963e-0a5efa640280-tmp-dir\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:40.901802 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:40.901624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68a8e641-6318-47ab-9605-39cc5dfb3c81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68a8e641-6318-47ab-9605-39cc5dfb3c81-crio-socket\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a8e641-6318-47ab-9605-39cc5dfb3c81-data-volume\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb53d9a8-99c6-4a02-963e-0a5efa640280-tmp-dir\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.002513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68a8e641-6318-47ab-9605-39cc5dfb3c81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb53d9a8-99c6-4a02-963e-0a5efa640280-config-volume\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.002513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3758b02-ff8d-4023-8e56-10f9d5e25be0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.002513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68a8e641-6318-47ab-9605-39cc5dfb3c81-crio-socket\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8gz\" (UniqueName: \"kubernetes.io/projected/a3758b02-ff8d-4023-8e56-10f9d5e25be0-kube-api-access-8g8gz\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a8e641-6318-47ab-9605-39cc5dfb3c81-data-volume\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb53d9a8-99c6-4a02-963e-0a5efa640280-tmp-dir\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8lw\" (UniqueName: \"kubernetes.io/projected/68a8e641-6318-47ab-9605-39cc5dfb3c81-kube-api-access-ch8lw\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb53d9a8-99c6-4a02-963e-0a5efa640280-metrics-tls\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7msr\" (UniqueName: \"kubernetes.io/projected/fb53d9a8-99c6-4a02-963e-0a5efa640280-kube-api-access-h7msr\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.002873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e27269-7607-4483-b8c3-b161b07da691-cert\") pod \"ingress-canary-d7jkb\" (UID: \"a8e27269-7607-4483-b8c3-b161b07da691\") " pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:41.003236 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68a8e641-6318-47ab-9605-39cc5dfb3c81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.003236 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.002928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwtg5\" (UniqueName: \"kubernetes.io/projected/a8e27269-7607-4483-b8c3-b161b07da691-kube-api-access-vwtg5\") pod \"ingress-canary-d7jkb\" (UID: \"a8e27269-7607-4483-b8c3-b161b07da691\") " pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:41.003236 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.003072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68a8e641-6318-47ab-9605-39cc5dfb3c81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.003236 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.003116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb53d9a8-99c6-4a02-963e-0a5efa640280-config-volume\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.003236 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.003167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3758b02-ff8d-4023-8e56-10f9d5e25be0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.003478 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:41.003459    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 13:31:41.003538 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:41.003524    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-tls podName:a3758b02-ff8d-4023-8e56-10f9d5e25be0 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:41.503510973 +0000 UTC m=+45.528325654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-5dd89" (UID: "a3758b02-ff8d-4023-8e56-10f9d5e25be0") : secret "prometheus-operator-tls" not found
Sep 10 13:31:41.006963 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.006868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e27269-7607-4483-b8c3-b161b07da691-cert\") pod \"ingress-canary-d7jkb\" (UID: \"a8e27269-7607-4483-b8c3-b161b07da691\") " pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:41.007044 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.006882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.007044 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.006882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb53d9a8-99c6-4a02-963e-0a5efa640280-metrics-tls\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.007044 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.006898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68a8e641-6318-47ab-9605-39cc5dfb3c81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.015478 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.015454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7msr\" (UniqueName: \"kubernetes.io/projected/fb53d9a8-99c6-4a02-963e-0a5efa640280-kube-api-access-h7msr\") pod \"dns-default-p2xt2\" (UID: \"fb53d9a8-99c6-4a02-963e-0a5efa640280\") " pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.015891 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.015872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8lw\" (UniqueName: \"kubernetes.io/projected/68a8e641-6318-47ab-9605-39cc5dfb3c81-kube-api-access-ch8lw\") pod \"insights-runtime-extractor-8zp5t\" (UID: \"68a8e641-6318-47ab-9605-39cc5dfb3c81\") " pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.016863 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.016843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8gz\" (UniqueName: \"kubernetes.io/projected/a3758b02-ff8d-4023-8e56-10f9d5e25be0-kube-api-access-8g8gz\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.018589 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.018552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwtg5\" (UniqueName: \"kubernetes.io/projected/a8e27269-7607-4483-b8c3-b161b07da691-kube-api-access-vwtg5\") pod \"ingress-canary-d7jkb\" (UID: \"a8e27269-7607-4483-b8c3-b161b07da691\") " pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:41.147047 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.147014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d7jkb"
Sep 10 13:31:41.153182 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.153166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zp5t"
Sep 10 13:31:41.163334 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.163312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:41.446051 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.446028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:31:41.446724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.446028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:31:41.449113 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.449093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twwrf\""
Sep 10 13:31:41.449237 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.449131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 13:31:41.449314 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.449265    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 13:31:41.449373 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.449346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 13:31:41.490876 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.490854    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d7jkb"]
Sep 10 13:31:41.494153 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:41.494129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e27269_7607_4483_b8c3_b161b07da691.slice/crio-934803b660f58115f3f98759e09c481ad7efec80272db2dfd1e78e6ef1ac3ede WatchSource:0}: Error finding container 934803b660f58115f3f98759e09c481ad7efec80272db2dfd1e78e6ef1ac3ede: Status 404 returned error can't find the container with id 934803b660f58115f3f98759e09c481ad7efec80272db2dfd1e78e6ef1ac3ede
Sep 10 13:31:41.494249 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.494162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zp5t"]
Sep 10 13:31:41.496053 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:41.496033    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a8e641_6318_47ab_9605_39cc5dfb3c81.slice/crio-4ed0d8712cf1c5c3f7c971c501c98542b2e905eae91dd766a2a9c07408734d61 WatchSource:0}: Error finding container 4ed0d8712cf1c5c3f7c971c501c98542b2e905eae91dd766a2a9c07408734d61: Status 404 returned error can't find the container with id 4ed0d8712cf1c5c3f7c971c501c98542b2e905eae91dd766a2a9c07408734d61
Sep 10 13:31:41.506189 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.506148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.508902 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.508880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2xt2"]
Sep 10 13:31:41.508993 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.508936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3758b02-ff8d-4023-8e56-10f9d5e25be0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-5dd89\" (UID: \"a3758b02-ff8d-4023-8e56-10f9d5e25be0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:41.511479 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:41.511461    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb53d9a8_99c6_4a02_963e_0a5efa640280.slice/crio-279f3568e9f633f37cf2d44ec3ac69e8ca506086826da2549daa3eec0ebe95ce WatchSource:0}: Error finding container 279f3568e9f633f37cf2d44ec3ac69e8ca506086826da2549daa3eec0ebe95ce: Status 404 returned error can't find the container with id 279f3568e9f633f37cf2d44ec3ac69e8ca506086826da2549daa3eec0ebe95ce
Sep 10 13:31:41.620091 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.620057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2xt2" event={"ID":"fb53d9a8-99c6-4a02-963e-0a5efa640280","Type":"ContainerStarted","Data":"279f3568e9f633f37cf2d44ec3ac69e8ca506086826da2549daa3eec0ebe95ce"}
Sep 10 13:31:41.621436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.621411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zp5t" event={"ID":"68a8e641-6318-47ab-9605-39cc5dfb3c81","Type":"ContainerStarted","Data":"5fd5ab3364c6012d33047394cccf6e134b0ca8501020bf9b4ab4c93ca300c0b0"}
Sep 10 13:31:41.621540 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.621490    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zp5t" event={"ID":"68a8e641-6318-47ab-9605-39cc5dfb3c81","Type":"ContainerStarted","Data":"4ed0d8712cf1c5c3f7c971c501c98542b2e905eae91dd766a2a9c07408734d61"}
Sep 10 13:31:41.622698 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.622658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d7jkb" event={"ID":"a8e27269-7607-4483-b8c3-b161b07da691","Type":"ContainerStarted","Data":"934803b660f58115f3f98759e09c481ad7efec80272db2dfd1e78e6ef1ac3ede"}
Sep 10 13:31:41.740848 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:41.740817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89"
Sep 10 13:31:42.028306 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:42.028051    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-5dd89"]
Sep 10 13:31:42.032449 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:42.032429    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3758b02_ff8d_4023_8e56_10f9d5e25be0.slice/crio-4747784df46e6deaa84796e09523f86c1146d743720cb40176d38304b3caa3b7 WatchSource:0}: Error finding container 4747784df46e6deaa84796e09523f86c1146d743720cb40176d38304b3caa3b7: Status 404 returned error can't find the container with id 4747784df46e6deaa84796e09523f86c1146d743720cb40176d38304b3caa3b7
Sep 10 13:31:42.446597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:42.446556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:31:42.450920 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:42.450827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 13:31:42.450920 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:42.450827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nc4jc\""
Sep 10 13:31:42.631002 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:42.630921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89" event={"ID":"a3758b02-ff8d-4023-8e56-10f9d5e25be0","Type":"ContainerStarted","Data":"4747784df46e6deaa84796e09523f86c1146d743720cb40176d38304b3caa3b7"}
Sep 10 13:31:43.635015 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.634970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zp5t" event={"ID":"68a8e641-6318-47ab-9605-39cc5dfb3c81","Type":"ContainerStarted","Data":"af8a4844b5edda3501976be21898332a7aa8bcc90d0a47fe5485d6b62a6a23e8"}
Sep 10 13:31:43.807636 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.807601    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8545c46595-28ndv"]
Sep 10 13:31:43.826092 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.826065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:43.828062 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.828037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8545c46595-28ndv"]
Sep 10 13:31:43.828998 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.828968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 13:31:43.829177 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.829017    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 13:31:43.829177 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.829023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 13:31:43.829307 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.829196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 13:31:43.829887 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.829801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 13:31:43.830043 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.830016    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 13:31:43.830043 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.830031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 13:31:43.830283 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.830261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5dh5t\""
Sep 10 13:31:43.929046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.928972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-config\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:43.929046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.929041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-oauth-config\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:43.929241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.929072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-service-ca\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:43.929241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.929095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-serving-cert\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:43.929241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.929120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr77t\" (UniqueName: \"kubernetes.io/projected/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-kube-api-access-cr77t\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:43.929241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:43.929197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-oauth-serving-cert\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.030221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.030187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-oauth-config\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.030377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.030231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-service-ca\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.030377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.030256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-serving-cert\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.030377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.030282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr77t\" (UniqueName: \"kubernetes.io/projected/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-kube-api-access-cr77t\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.030896 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.030713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-oauth-serving-cert\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.030896 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.030810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-config\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.031324 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.031096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-service-ca\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.031324 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.031310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-oauth-serving-cert\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.031496 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.031468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-config\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.033913 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.033882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-oauth-config\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.033995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.033924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-serving-cert\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.055212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.055187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr77t\" (UniqueName: \"kubernetes.io/projected/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-kube-api-access-cr77t\") pod \"console-8545c46595-28ndv\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") " pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:44.138809 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:44.138778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:45.848983 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:45.848660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8545c46595-28ndv"]
Sep 10 13:31:45.857334 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:45.857295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc661eb4_1600_4c53_9adb_c6ea79c73b1a.slice/crio-fc708e2a075f8dc103e8524bd6757cf9b1774cc8aadad53238ee08310c0d791b WatchSource:0}: Error finding container fc708e2a075f8dc103e8524bd6757cf9b1774cc8aadad53238ee08310c0d791b: Status 404 returned error can't find the container with id fc708e2a075f8dc103e8524bd6757cf9b1774cc8aadad53238ee08310c0d791b
Sep 10 13:31:46.645658 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.645613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8545c46595-28ndv" event={"ID":"cc661eb4-1600-4c53-9adb-c6ea79c73b1a","Type":"ContainerStarted","Data":"fc708e2a075f8dc103e8524bd6757cf9b1774cc8aadad53238ee08310c0d791b"}
Sep 10 13:31:46.647686 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.647654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2xt2" event={"ID":"fb53d9a8-99c6-4a02-963e-0a5efa640280","Type":"ContainerStarted","Data":"aea53863a64260fe86edac8e936718e2b55344f3d81244d7529694f3cf2e69ae"}
Sep 10 13:31:46.647806 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.647713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2xt2" event={"ID":"fb53d9a8-99c6-4a02-963e-0a5efa640280","Type":"ContainerStarted","Data":"80d425fe10a5026199d8d4cc768d88c05e070f347d976ce0082a46e120e15bd9"}
Sep 10 13:31:46.647881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.647797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:46.650089 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.650062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89" event={"ID":"a3758b02-ff8d-4023-8e56-10f9d5e25be0","Type":"ContainerStarted","Data":"7c4025e768b9cf48e854ab8b434db1e7acde8dccab7ca1a3dd282963452297c7"}
Sep 10 13:31:46.650200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.650110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89" event={"ID":"a3758b02-ff8d-4023-8e56-10f9d5e25be0","Type":"ContainerStarted","Data":"7d2500e5d22d3531d248c32fa1ba275ed1acc0c1a270dbcda4941b2893190524"}
Sep 10 13:31:46.651687 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.651661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d7jkb" event={"ID":"a8e27269-7607-4483-b8c3-b161b07da691","Type":"ContainerStarted","Data":"09ed9baa354500c10b7df9cd05223a28aa143146e57ded92df244e8896303cbc"}
Sep 10 13:31:46.669723 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.669679    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p2xt2" podStartSLOduration=2.617252499 podStartE2EDuration="6.669666643s" totalImagesPullingTime="4.052414144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:46.668738963 +0000 UTC m=+50.693553672" watchObservedRunningTime="2026-09-10 13:31:46.669666643 +0000 UTC m=+50.694481337"
Sep 10 13:31:46.705755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.705710    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-5dd89" podStartSLOduration=3.175623469 podStartE2EDuration="6.70569411s" totalImagesPullingTime="3.530070641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:46.68855374 +0000 UTC m=+50.713368443" watchObservedRunningTime="2026-09-10 13:31:46.70569411 +0000 UTC m=+50.730508817"
Sep 10 13:31:46.706034 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:46.706006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d7jkb" podStartSLOduration=2.630756838 podStartE2EDuration="6.705996904s" totalImagesPullingTime="4.075240066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:46.704960376 +0000 UTC m=+50.729775080" watchObservedRunningTime="2026-09-10 13:31:46.705996904 +0000 UTC m=+50.730811607"
Sep 10 13:31:48.456634 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.456435    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"]
Sep 10 13:31:48.458779 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.458762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.461788 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.461744    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 13:31:48.461904 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.461850    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bm697\""
Sep 10 13:31:48.462377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.462273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 13:31:48.511613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.511570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"]
Sep 10 13:31:48.548765 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.548739    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"]
Sep 10 13:31:48.551286 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.551263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.556782 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.556756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hnzj7\""
Sep 10 13:31:48.556892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.556811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 13:31:48.556946 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.556920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 13:31:48.557849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.557831    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 13:31:48.564834 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.564811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1379c126-248e-4ca8-ae92-091740b115b2-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.564937 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.564907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.564999 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.564938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.565054 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.565003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88cw\" (UniqueName: \"kubernetes.io/projected/1379c126-248e-4ca8-ae92-091740b115b2-kube-api-access-s88cw\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.577910 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.577871    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"]
Sep 10 13:31:48.665856 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.665827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d089c83-40a2-4b68-b18f-fd20f010483d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.666066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.665875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.666066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.665961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.666066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.666185 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88cw\" (UniqueName: \"kubernetes.io/projected/1379c126-248e-4ca8-ae92-091740b115b2-kube-api-access-s88cw\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.666185 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1379c126-248e-4ca8-ae92-091740b115b2-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.666185 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:48.666123    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 13:31:48.666185 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d089c83-40a2-4b68-b18f-fd20f010483d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.666185 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mw8\" (UniqueName: \"kubernetes.io/projected/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-api-access-l6mw8\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.666185 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.666381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.666196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.666609 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:48.666591    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls podName:1379c126-248e-4ca8-ae92-091740b115b2 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:49.166556491 +0000 UTC m=+53.191371177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-gvp4m" (UID: "1379c126-248e-4ca8-ae92-091740b115b2") : secret "openshift-state-metrics-tls" not found
Sep 10 13:31:48.667261 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.667243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1379c126-248e-4ca8-ae92-091740b115b2-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.670384 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.670359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.676325 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.676302    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c98nt"]
Sep 10 13:31:48.679684 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.679663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.684151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.684128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 13:31:48.684226 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.684165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tt9jm\""
Sep 10 13:31:48.684525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.684446    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 13:31:48.684657 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.684543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 13:31:48.693267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.693235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88cw\" (UniqueName: \"kubernetes.io/projected/1379c126-248e-4ca8-ae92-091740b115b2-kube-api-access-s88cw\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:48.767337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-textfile\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.767337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-sys\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767628 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.767628 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767628 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-root\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767628 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68wz\" (UniqueName: \"kubernetes.io/projected/1be5bf26-1e6a-40be-a453-990bfb64e7c7-kube-api-access-b68wz\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767835 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1be5bf26-1e6a-40be-a453-990bfb64e7c7-metrics-client-ca\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767835 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.767835 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d089c83-40a2-4b68-b18f-fd20f010483d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.767835 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mw8\" (UniqueName: \"kubernetes.io/projected/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-api-access-l6mw8\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.767835 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.767808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.768146 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.768120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d089c83-40a2-4b68-b18f-fd20f010483d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.768238 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.768149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-tls\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.768238 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.768205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d089c83-40a2-4b68-b18f-fd20f010483d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.768238 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.768235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-wtmp\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.768381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.768326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.768959 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.768934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d089c83-40a2-4b68-b18f-fd20f010483d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.770695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.770653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.770801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.770767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.793995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.793968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mw8\" (UniqueName: \"kubernetes.io/projected/0d089c83-40a2-4b68-b18f-fd20f010483d-kube-api-access-l6mw8\") pod \"kube-state-metrics-74cb57df7-2snc9\" (UID: \"0d089c83-40a2-4b68-b18f-fd20f010483d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.862118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.862078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"
Sep 10 13:31:48.869371 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1be5bf26-1e6a-40be-a453-990bfb64e7c7-metrics-client-ca\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-tls\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-wtmp\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-textfile\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-sys\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869746 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869746 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-root\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869746 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-wtmp\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869746 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wz\" (UniqueName: \"kubernetes.io/projected/1be5bf26-1e6a-40be-a453-990bfb64e7c7-kube-api-access-b68wz\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.869921 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.869797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-sys\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.870062 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.870037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1be5bf26-1e6a-40be-a453-990bfb64e7c7-metrics-client-ca\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.870062 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.870054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-textfile\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.870206 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.870080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1be5bf26-1e6a-40be-a453-990bfb64e7c7-root\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.870448 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.870424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.872418 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.872397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-tls\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.872735 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.872715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1be5bf26-1e6a-40be-a453-990bfb64e7c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.882977 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.882953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68wz\" (UniqueName: \"kubernetes.io/projected/1be5bf26-1e6a-40be-a453-990bfb64e7c7-kube-api-access-b68wz\") pod \"node-exporter-c98nt\" (UID: \"1be5bf26-1e6a-40be-a453-990bfb64e7c7\") " pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:48.990760 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:48.990732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c98nt"
Sep 10 13:31:49.149385 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:49.149357    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be5bf26_1e6a_40be_a453_990bfb64e7c7.slice/crio-df885953049a0aaecdead31d914d883a50f042b19bf065071bd75dbae82232e8 WatchSource:0}: Error finding container df885953049a0aaecdead31d914d883a50f042b19bf065071bd75dbae82232e8: Status 404 returned error can't find the container with id df885953049a0aaecdead31d914d883a50f042b19bf065071bd75dbae82232e8
Sep 10 13:31:49.171810 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.171785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:49.171916 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:49.171902    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 13:31:49.171982 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:49.171959    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls podName:1379c126-248e-4ca8-ae92-091740b115b2 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:50.171945985 +0000 UTC m=+54.196760667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-gvp4m" (UID: "1379c126-248e-4ca8-ae92-091740b115b2") : secret "openshift-state-metrics-tls" not found
Sep 10 13:31:49.441378 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.441304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-2snc9"]
Sep 10 13:31:49.447950 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:49.447920    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d089c83_40a2_4b68_b18f_fd20f010483d.slice/crio-df3479b794d9d66a936296529a4ec19f90698698a3f1ee8e4898f0cff6ff9e89 WatchSource:0}: Error finding container df3479b794d9d66a936296529a4ec19f90698698a3f1ee8e4898f0cff6ff9e89: Status 404 returned error can't find the container with id df3479b794d9d66a936296529a4ec19f90698698a3f1ee8e4898f0cff6ff9e89
Sep 10 13:31:49.528353 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.528319    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:31:49.530980 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.530961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.538713 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.538693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 13:31:49.538713 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.538711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tknhq\""
Sep 10 13:31:49.539246 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539231    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 13:31:49.539302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539270    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 13:31:49.539636 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 13:31:49.539721 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 13:31:49.539721 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 13:31:49.539945 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539931    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 13:31:49.540000 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.539990    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 13:31:49.548738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.548723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 13:31:49.560398 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.560378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:31:49.575168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575276 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-out\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575276 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575276 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxkz\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-kube-api-access-fkxkz\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575378 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575378 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575472 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-web-config\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575516 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-volume\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575600 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575633 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.575671 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.575646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.664434 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.664394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9" event={"ID":"0d089c83-40a2-4b68-b18f-fd20f010483d","Type":"ContainerStarted","Data":"df3479b794d9d66a936296529a4ec19f90698698a3f1ee8e4898f0cff6ff9e89"}
Sep 10 13:31:49.666488 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.666454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zp5t" event={"ID":"68a8e641-6318-47ab-9605-39cc5dfb3c81","Type":"ContainerStarted","Data":"07f948666a40ce983ace31a7bd42c7ef4f3957fe477001bbc6e1413b3c6fe48f"}
Sep 10 13:31:49.668118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.668087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8545c46595-28ndv" event={"ID":"cc661eb4-1600-4c53-9adb-c6ea79c73b1a","Type":"ContainerStarted","Data":"930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf"}
Sep 10 13:31:49.669424 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.669404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c98nt" event={"ID":"1be5bf26-1e6a-40be-a453-990bfb64e7c7","Type":"ContainerStarted","Data":"df885953049a0aaecdead31d914d883a50f042b19bf065071bd75dbae82232e8"}
Sep 10 13:31:49.676472 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676554 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-out\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676554 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxkz\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-kube-api-access-fkxkz\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-web-config\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-volume\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676940 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676940 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676940 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.676940 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.676902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.677148 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.677038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.677205 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:49.677166    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle podName:83c3e51c-3958-47e2-9063-c4a084fceaa0 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:50.177145885 +0000 UTC m=+54.201960569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0") : configmap references non-existent config key: ca-bundle.crt
Sep 10 13:31:49.680362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.680337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.681026 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.681003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.681491 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.681465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.681625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.681602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.681625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.681610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.681752 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.681613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-web-config\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.681752 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.681706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.682439 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.682397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-out\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.682564 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.682532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.682655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.682633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-volume\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.700509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.700452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxkz\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-kube-api-access-fkxkz\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:49.708130 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.708089    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8zp5t" podStartSLOduration=2.324907211 podStartE2EDuration="9.708076039s" totalImagesPullingTime="7.383168828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:49.707918793 +0000 UTC m=+53.732733495" watchObservedRunningTime="2026-09-10 13:31:49.708076039 +0000 UTC m=+53.732890736"
Sep 10 13:31:49.765420 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:49.765375    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8545c46595-28ndv" podStartSLOduration=3.477264954 podStartE2EDuration="6.765361574s" totalImagesPullingTime="3.28809662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:49.764465389 +0000 UTC m=+53.789280092" watchObservedRunningTime="2026-09-10 13:31:49.765361574 +0000 UTC m=+53.790176276"
Sep 10 13:31:50.180313 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.180267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:50.180517 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.180353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:50.181386 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.181358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:50.183026 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.182998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1379c126-248e-4ca8-ae92-091740b115b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gvp4m\" (UID: \"1379c126-248e-4ca8-ae92-091740b115b2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:50.272644 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.272610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"
Sep 10 13:31:50.440399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.440324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:31:50.492393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.492108    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b7665cccd-249vg"]
Sep 10 13:31:50.495164 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.495149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.499738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.499703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 13:31:50.499967 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.499948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 13:31:50.500137 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.500094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4gqkp\""
Sep 10 13:31:50.500215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.500175    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 13:31:50.500704 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.500686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 13:31:50.500962 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.500880    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a7h4k8c8ud6v2\""
Sep 10 13:31:50.503042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.503023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/308713cb-bd60-4b14-9b27-5c4e700091da-metrics-client-ca\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv98r\" (UniqueName: \"kubernetes.io/projected/308713cb-bd60-4b14-9b27-5c4e700091da-kube-api-access-kv98r\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.583985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-grpc-tls\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.584239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.584064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-tls\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.585497 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.585461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m"]
Sep 10 13:31:50.587815 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:50.587791    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1379c126_248e_4ca8_ae92_091740b115b2.slice/crio-4c27565f186069892c5a39467865fd03f749bce5e3a5953f990fc340c6c16cc7 WatchSource:0}: Error finding container 4c27565f186069892c5a39467865fd03f749bce5e3a5953f990fc340c6c16cc7: Status 404 returned error can't find the container with id 4c27565f186069892c5a39467865fd03f749bce5e3a5953f990fc340c6c16cc7
Sep 10 13:31:50.623239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.623206    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b7665cccd-249vg"]
Sep 10 13:31:50.682162 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.682131    2577 generic.go:356] "Generic (PLEG): container finished" podID="1be5bf26-1e6a-40be-a453-990bfb64e7c7" containerID="27d665f49864a9ea71abc40633f0a95ba66c376ebda10dd12ccbf8e426f8fadd" exitCode=0
Sep 10 13:31:50.682394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.682216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c98nt" event={"ID":"1be5bf26-1e6a-40be-a453-990bfb64e7c7","Type":"ContainerDied","Data":"27d665f49864a9ea71abc40633f0a95ba66c376ebda10dd12ccbf8e426f8fadd"}
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m" event={"ID":"1379c126-248e-4ca8-ae92-091740b115b2","Type":"ContainerStarted","Data":"2311c519949b038b5bb5135f3946f2c3ad1a9b8e68c3bfe71d81de45349362ee"}
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m" event={"ID":"1379c126-248e-4ca8-ae92-091740b115b2","Type":"ContainerStarted","Data":"4c27565f186069892c5a39467865fd03f749bce5e3a5953f990fc340c6c16cc7"}
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-tls\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/308713cb-bd60-4b14-9b27-5c4e700091da-metrics-client-ca\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.685726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv98r\" (UniqueName: \"kubernetes.io/projected/308713cb-bd60-4b14-9b27-5c4e700091da-kube-api-access-kv98r\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.686104 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.686104 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.686104 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.685803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-grpc-tls\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.691664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.688634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/308713cb-bd60-4b14-9b27-5c4e700091da-metrics-client-ca\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.691664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.689776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.691664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.691468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.692100 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.692033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-tls\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.692246 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.692192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.692922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.692873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.693284 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.693245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/308713cb-bd60-4b14-9b27-5c4e700091da-secret-grpc-tls\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.710748 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.710725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv98r\" (UniqueName: \"kubernetes.io/projected/308713cb-bd60-4b14-9b27-5c4e700091da-kube-api-access-kv98r\") pod \"thanos-querier-5b7665cccd-249vg\" (UID: \"308713cb-bd60-4b14-9b27-5c4e700091da\") " pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:50.750122 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.750093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:31:50.754980 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:50.754951    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c3e51c_3958_47e2_9063_c4a084fceaa0.slice/crio-e0e7bbebcbba603c78ba8db9abfcd84129149e19a37e7223361dedbe458ef69b WatchSource:0}: Error finding container e0e7bbebcbba603c78ba8db9abfcd84129149e19a37e7223361dedbe458ef69b: Status 404 returned error can't find the container with id e0e7bbebcbba603c78ba8db9abfcd84129149e19a37e7223361dedbe458ef69b
Sep 10 13:31:50.837549 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:50.837525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:51.125540 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.125510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b7665cccd-249vg"]
Sep 10 13:31:51.178057 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:51.178027    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308713cb_bd60_4b14_9b27_5c4e700091da.slice/crio-feb74f08e1fd3df959a04c5366b0b060ab04bb003241f817909316eb13ec8a2b WatchSource:0}: Error finding container feb74f08e1fd3df959a04c5366b0b060ab04bb003241f817909316eb13ec8a2b: Status 404 returned error can't find the container with id feb74f08e1fd3df959a04c5366b0b060ab04bb003241f817909316eb13ec8a2b
Sep 10 13:31:51.682370 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.682286    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7557cdb568-d4bnr"]
Sep 10 13:31:51.686018 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.685215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.699340 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.699310    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 13:31:51.699930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.699899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m" event={"ID":"1379c126-248e-4ca8-ae92-091740b115b2","Type":"ContainerStarted","Data":"8ea44a0f2b3fde9a51ad586e09a66b25364d604b547481523d6ee1784bf7fc0e"}
Sep 10 13:31:51.704149 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.704123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9" event={"ID":"0d089c83-40a2-4b68-b18f-fd20f010483d","Type":"ContainerStarted","Data":"5102434f1ab77da80266780d1d1afbdc7f78df3414b1c694e56ceb860ef36d12"}
Sep 10 13:31:51.704247 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.704178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9" event={"ID":"0d089c83-40a2-4b68-b18f-fd20f010483d","Type":"ContainerStarted","Data":"76f7a18ff7e500728bb74c3c4185666e3603e37a9b9b6ee0adf898923b4c21af"}
Sep 10 13:31:51.704247 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.704198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9" event={"ID":"0d089c83-40a2-4b68-b18f-fd20f010483d","Type":"ContainerStarted","Data":"e1ca0d4124c68415c99c60bfc4a3971e31bbaadbde261ecefd15c5e29b8300db"}
Sep 10 13:31:51.705951 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.705891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"feb74f08e1fd3df959a04c5366b0b060ab04bb003241f817909316eb13ec8a2b"}
Sep 10 13:31:51.708564 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.708540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c98nt" event={"ID":"1be5bf26-1e6a-40be-a453-990bfb64e7c7","Type":"ContainerStarted","Data":"2d5849158361587d241522b17e5d7c46675cf31807feda1a5935a7c6dea58b24"}
Sep 10 13:31:51.708672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.708609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c98nt" event={"ID":"1be5bf26-1e6a-40be-a453-990bfb64e7c7","Type":"ContainerStarted","Data":"ac989fe9b95fd0549efba509d9bd973337f3f6928c35937ef6fa59026a2446c3"}
Sep 10 13:31:51.711759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.711735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"e0e7bbebcbba603c78ba8db9abfcd84129149e19a37e7223361dedbe458ef69b"}
Sep 10 13:31:51.712469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.712450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7557cdb568-d4bnr"]
Sep 10 13:31:51.777606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.777308    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2snc9" podStartSLOduration=2.018222633 podStartE2EDuration="3.777287916s" totalImagesPullingTime="1.759065283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:51.776835485 +0000 UTC m=+55.801650190" watchObservedRunningTime="2026-09-10 13:31:51.777287916 +0000 UTC m=+55.802102615"
Sep 10 13:31:51.796008 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.795962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-oauth-serving-cert\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.796293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.796131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-serving-cert\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.796293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.796203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-oauth-config\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.796293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.796230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-config\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.796293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.796256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-trusted-ca-bundle\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.796517 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.796312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-service-ca\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.796517 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.796400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmhn\" (UniqueName: \"kubernetes.io/projected/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-kube-api-access-sgmhn\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.824843 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.824790    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c98nt" podStartSLOduration=2.61785045 podStartE2EDuration="3.82477137s" totalImagesPullingTime="1.143412799s" totalInitContainerRuntime="63.508121ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:51.824508455 +0000 UTC m=+55.849323159" watchObservedRunningTime="2026-09-10 13:31:51.82477137 +0000 UTC m=+55.849586075"
Sep 10 13:31:51.897331 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-oauth-serving-cert\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.897526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-serving-cert\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.897526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-oauth-config\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.897526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-config\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.897526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-trusted-ca-bundle\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.897526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-service-ca\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.897526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.897500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmhn\" (UniqueName: \"kubernetes.io/projected/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-kube-api-access-sgmhn\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.899072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.898690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-oauth-serving-cert\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.899072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.898690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-config\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.899072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.899013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-trusted-ca-bundle\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.899295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.899274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-service-ca\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.900769 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.900747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-oauth-config\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.901429 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.901411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-serving-cert\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:51.909529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:51.909508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmhn\" (UniqueName: \"kubernetes.io/projected/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-kube-api-access-sgmhn\") pod \"console-7557cdb568-d4bnr\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") " pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:52.003936 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.003847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:31:52.481888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.481857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7557cdb568-d4bnr"]
Sep 10 13:31:52.485809 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:52.485782    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec7c1d0_86fa_4915_a7b7_ed5b4925b12d.slice/crio-b7219e638c845fb191c53113898cec3c9f437ff468ff8c3f0f1a47d6085386bc WatchSource:0}: Error finding container b7219e638c845fb191c53113898cec3c9f437ff468ff8c3f0f1a47d6085386bc: Status 404 returned error can't find the container with id b7219e638c845fb191c53113898cec3c9f437ff468ff8c3f0f1a47d6085386bc
Sep 10 13:31:52.717771 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.717730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7557cdb568-d4bnr" event={"ID":"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d","Type":"ContainerStarted","Data":"47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4"}
Sep 10 13:31:52.718196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.717795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7557cdb568-d4bnr" event={"ID":"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d","Type":"ContainerStarted","Data":"b7219e638c845fb191c53113898cec3c9f437ff468ff8c3f0f1a47d6085386bc"}
Sep 10 13:31:52.719528 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.719497    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="5d7bcb514e45f155aeaef1c7ec3720abffcc37bb3bc656f2a008160cfcc8b45e" exitCode=0
Sep 10 13:31:52.719675 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.719614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"5d7bcb514e45f155aeaef1c7ec3720abffcc37bb3bc656f2a008160cfcc8b45e"}
Sep 10 13:31:52.722368 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.722310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m" event={"ID":"1379c126-248e-4ca8-ae92-091740b115b2","Type":"ContainerStarted","Data":"67a456166416524c5ed4a1eb444e482b36db30da1a5a72efbe8d93e0d1bcd370"}
Sep 10 13:31:52.754057 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.754012    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7557cdb568-d4bnr" podStartSLOduration=1.7539948920000001 podStartE2EDuration="1.753994892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:52.753328627 +0000 UTC m=+56.778143333" watchObservedRunningTime="2026-09-10 13:31:52.753994892 +0000 UTC m=+56.778809591"
Sep 10 13:31:52.790623 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:52.790494    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gvp4m" podStartSLOduration=3.367603108 podStartE2EDuration="4.790480301s" totalImagesPullingTime="1.422877193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:52.789947642 +0000 UTC m=+56.814762355" watchObservedRunningTime="2026-09-10 13:31:52.790480301 +0000 UTC m=+56.815295005"
Sep 10 13:31:53.092446 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.092413    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"]
Sep 10 13:31:53.094841 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.094819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.098124 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.098092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 13:31:53.098253 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.098132    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vsfxm\""
Sep 10 13:31:53.134429 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.134400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"]
Sep 10 13:31:53.215539 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.215501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswf4\" (UniqueName: \"kubernetes.io/projected/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-kube-api-access-jswf4\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.215750 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.215708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.222548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.222522    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fc48648d6-rpkvd"]
Sep 10 13:31:53.226597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.226565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.229445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.229412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 13:31:53.229555 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.229453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 13:31:53.229683 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.229647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-blwqn\""
Sep 10 13:31:53.229778 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.229724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 13:31:53.233686 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.233657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 13:31:53.235509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.235487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8srh5h9fre40i\""
Sep 10 13:31:53.243800 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.243778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fc48648d6-rpkvd"]
Sep 10 13:31:53.316635 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.316602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswf4\" (UniqueName: \"kubernetes.io/projected/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-kube-api-access-jswf4\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.316827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.316731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-client-ca-bundle\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.316827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.316805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b521937-f3ce-4996-94db-e978d886b5d3-metrics-server-audit-profiles\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.316959 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.316849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.316959 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.316924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b521937-f3ce-4996-94db-e978d886b5d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.317070 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:53.316957    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 13:31:53.317070 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:53.317017    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert podName:ef1f53f6-423a-434f-aa64-ef3f2d4e96f2 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:53.816998599 +0000 UTC m=+57.841813297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-zrpv9" (UID: "ef1f53f6-423a-434f-aa64-ef3f2d4e96f2") : secret "monitoring-plugin-cert" not found
Sep 10 13:31:53.317070 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.317039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7nd\" (UniqueName: \"kubernetes.io/projected/2b521937-f3ce-4996-94db-e978d886b5d3-kube-api-access-5z7nd\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.317070 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.317070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-secret-metrics-server-tls\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.317279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.317164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-secret-metrics-server-client-certs\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.317279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.317234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b521937-f3ce-4996-94db-e978d886b5d3-audit-log\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.335464 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.335432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswf4\" (UniqueName: \"kubernetes.io/projected/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-kube-api-access-jswf4\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.417790 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.417711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b521937-f3ce-4996-94db-e978d886b5d3-audit-log\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.417790 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.417774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-client-ca-bundle\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.417825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b521937-f3ce-4996-94db-e978d886b5d3-metrics-server-audit-profiles\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.417884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b521937-f3ce-4996-94db-e978d886b5d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.417932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7nd\" (UniqueName: \"kubernetes.io/projected/2b521937-f3ce-4996-94db-e978d886b5d3-kube-api-access-5z7nd\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.417956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-secret-metrics-server-tls\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418012 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.418008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-secret-metrics-server-client-certs\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418313 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.418193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b521937-f3ce-4996-94db-e978d886b5d3-audit-log\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.418948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b521937-f3ce-4996-94db-e978d886b5d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.418976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.418966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b521937-f3ce-4996-94db-e978d886b5d3-metrics-server-audit-profiles\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.420784 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.420760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-client-ca-bundle\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.421304 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.421278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-secret-metrics-server-client-certs\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.421402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.421327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b521937-f3ce-4996-94db-e978d886b5d3-secret-metrics-server-tls\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.441859 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.441829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7nd\" (UniqueName: \"kubernetes.io/projected/2b521937-f3ce-4996-94db-e978d886b5d3-kube-api-access-5z7nd\") pod \"metrics-server-7fc48648d6-rpkvd\" (UID: \"2b521937-f3ce-4996-94db-e978d886b5d3\") " pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.537368 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.537332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:31:53.823116 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.823085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:53.823505 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:53.823262    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 13:31:53.823505 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:31:53.823348    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert podName:ef1f53f6-423a-434f-aa64-ef3f2d4e96f2 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:54.823327963 +0000 UTC m=+58.848142658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-zrpv9" (UID: "ef1f53f6-423a-434f-aa64-ef3f2d4e96f2") : secret "monitoring-plugin-cert" not found
Sep 10 13:31:53.866236 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:53.866088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fc48648d6-rpkvd"]
Sep 10 13:31:53.925180 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:53.925147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b521937_f3ce_4996_94db_e978d886b5d3.slice/crio-21609d245cd8bc38ae8dfd6fa3f315194c2174c758d94c2395074f81ecd5b17e WatchSource:0}: Error finding container 21609d245cd8bc38ae8dfd6fa3f315194c2174c758d94c2395074f81ecd5b17e: Status 404 returned error can't find the container with id 21609d245cd8bc38ae8dfd6fa3f315194c2174c758d94c2395074f81ecd5b17e
Sep 10 13:31:54.139730 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.139707    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:54.139824 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.139761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:54.144965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.144944    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:54.731241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.731203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"b52835dcf2ed3a8e4d71ebdc4d7ad973fe1156b3946c429b223ee41fc065a4d1"}
Sep 10 13:31:54.731394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.731280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"b63aec43e49f7c547982932ba9bd15c37e10fb4d492d3341e10a5d8af65a4cda"}
Sep 10 13:31:54.731394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.731311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"72fd13d5778af52c5f4b719ce9ed0d81cccf59e8885fc894df1e91d5583dde37"}
Sep 10 13:31:54.732440 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.732417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd" event={"ID":"2b521937-f3ce-4996-94db-e978d886b5d3","Type":"ContainerStarted","Data":"21609d245cd8bc38ae8dfd6fa3f315194c2174c758d94c2395074f81ecd5b17e"}
Sep 10 13:31:54.737204 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.737182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:31:54.832566 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.832528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:54.836078 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.836046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef1f53f6-423a-434f-aa64-ef3f2d4e96f2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrpv9\" (UID: \"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:54.880541 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.880426    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:31:54.897716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.897689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:54.906123 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.905921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:54.909341 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.909288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 13:31:54.909341 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.909524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 13:31:54.909341 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.909761    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 13:31:54.910231 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.910152    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.911968    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912258    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f43cgjb55jclu\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.912905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.913102    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.913278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 13:31:54.914033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.913451    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7vwfr\""
Sep 10 13:31:54.922971 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.922944    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 13:31:54.930220 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:54.930190    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 13:31:55.036719 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.036866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.036866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.036866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.036980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdxv\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-kube-api-access-jmdxv\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-web-config\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-config-out\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-config\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.037613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.037453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.138959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-config\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdxv\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-kube-api-access-jmdxv\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-web-config\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-config-out\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.139987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.139632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.141985 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.141956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.143826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.142066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.143826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.143522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.144473 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.144215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.144473 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.144276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.144473 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.144433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-config-out\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.145615 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.144968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.145615 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.145322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.145615 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.145545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.146305 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.146262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.148423 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.147531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-web-config\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.148423 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.147559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.148423 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.147987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.148423 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.148316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.148746 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.148522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.148919 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.148896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.149546 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.149501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.150293 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.150272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.151223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.150868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-config\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.164569 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.164545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdxv\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-kube-api-access-jmdxv\") pod \"prometheus-k8s-0\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.226791 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.226455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:31:55.237117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.237004    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"]
Sep 10 13:31:55.332768 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:55.332730    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1f53f6_423a_434f_aa64_ef3f2d4e96f2.slice/crio-0418567ed7443a51617ee3ab83cccc9c77b9475f3263f065bd81b577b9fbf918 WatchSource:0}: Error finding container 0418567ed7443a51617ee3ab83cccc9c77b9475f3263f065bd81b577b9fbf918: Status 404 returned error can't find the container with id 0418567ed7443a51617ee3ab83cccc9c77b9475f3263f065bd81b577b9fbf918
Sep 10 13:31:55.625995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.625919    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:31:55.739960 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.739921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"2a39c33f1b0469fdf631aae825757203fca91795792b76e54ddadd510bb5de42"}
Sep 10 13:31:55.740120 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.739999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"3acc08d6243b5f13339aacee27201d3433821217321249ff4ded8393f8adafa2"}
Sep 10 13:31:55.740120 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.740045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" event={"ID":"308713cb-bd60-4b14-9b27-5c4e700091da","Type":"ContainerStarted","Data":"31a3f9ad7d62994a58b5f3f6b0f0c73dcce5a54add010b047acf2fe6baaa6aaa"}
Sep 10 13:31:55.740120 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.740087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:31:55.741401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.741373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9" event={"ID":"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2","Type":"ContainerStarted","Data":"0418567ed7443a51617ee3ab83cccc9c77b9475f3263f065bd81b577b9fbf918"}
Sep 10 13:31:55.770509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:55.770440    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg" podStartSLOduration=1.8441476030000001 podStartE2EDuration="5.770420845s" totalImagesPullingTime="3.926273242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:55.768216869 +0000 UTC m=+59.793031572" watchObservedRunningTime="2026-09-10 13:31:55.770420845 +0000 UTC m=+59.795235549"
Sep 10 13:31:55.920465 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:31:55.920403    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43015a99_800c_4032_92b0_9efdb8cecb8b.slice/crio-0cd60b62b9ebd8951048face164649542803d39fd9ee77d8d7b38e95e81e1c88 WatchSource:0}: Error finding container 0cd60b62b9ebd8951048face164649542803d39fd9ee77d8d7b38e95e81e1c88: Status 404 returned error can't find the container with id 0cd60b62b9ebd8951048face164649542803d39fd9ee77d8d7b38e95e81e1c88
Sep 10 13:31:56.590654 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.590379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjdx"
Sep 10 13:31:56.658760 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.658736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p2xt2"
Sep 10 13:31:56.748437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.748373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd" event={"ID":"2b521937-f3ce-4996-94db-e978d886b5d3","Type":"ContainerStarted","Data":"7651e31e2f9ccbce2d4c1be2c8e56f1141531316c9f743c57a11d2f58876a9fc"}
Sep 10 13:31:56.750697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.750637    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99" exitCode=0
Sep 10 13:31:56.750844 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.750719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99"}
Sep 10 13:31:56.750844 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.750797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"0cd60b62b9ebd8951048face164649542803d39fd9ee77d8d7b38e95e81e1c88"}
Sep 10 13:31:56.756107 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.756083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"bafda016d2f414108f2bacabecd5303ad88df2dbb311fb9d46769e5fe7dd4d40"}
Sep 10 13:31:56.756212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.756152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"03900e71c765cb99775032b549a545d3d09db30f4ba14806ae709964c2c83d21"}
Sep 10 13:31:56.756212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.756182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"e9ef98f9e53e0aa0cc927577b032a35d3d7e0e089ac71c6acbd702ee6cc18ed4"}
Sep 10 13:31:56.756212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.756208    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"eeb1954bc2875a362cc61375dad3f7062219464c2073402fb2259da4f10feee7"}
Sep 10 13:31:56.756362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.756233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"05939605d06d386ced7b75bcb1fa86722b1060b8d740c42e95611839f2a41915"}
Sep 10 13:31:56.756362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.756271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerStarted","Data":"713092130a9bef5d4f4bcad9b219d2ca1cec08c6d09b3d1b3ceb0998923e5d5e"}
Sep 10 13:31:56.792111 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.792021    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd" podStartSLOduration=1.465617278 podStartE2EDuration="3.792005197s" totalImagesPullingTime="2.326387919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:56.790510138 +0000 UTC m=+60.815324842" watchObservedRunningTime="2026-09-10 13:31:56.792005197 +0000 UTC m=+60.816819890"
Sep 10 13:31:56.862147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:56.862091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.848060224 podStartE2EDuration="7.862076709s" totalImagesPullingTime="4.994024077s" totalInitContainerRuntime="19.992408ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:56.859073806 +0000 UTC m=+60.883888510" watchObservedRunningTime="2026-09-10 13:31:56.862076709 +0000 UTC m=+60.886891411"
Sep 10 13:31:57.239054 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:57.239032    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7557cdb568-d4bnr"]
Sep 10 13:31:57.761396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:57.761355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9" event={"ID":"ef1f53f6-423a-434f-aa64-ef3f2d4e96f2","Type":"ContainerStarted","Data":"c45e158e0df17a6a20dcd9d6889af28fb41e650ea5ebb7e1a932b10bec2189e0"}
Sep 10 13:31:57.761668 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:57.761648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:57.766764 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:57.766737    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9"
Sep 10 13:31:57.797180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:31:57.797132    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrpv9" podStartSLOduration=2.943344055 podStartE2EDuration="4.797118426s" totalImagesPullingTime="1.853774371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:57.796198493 +0000 UTC m=+61.821013197" watchObservedRunningTime="2026-09-10 13:31:57.797118426 +0000 UTC m=+61.821933160"
Sep 10 13:32:01.763241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:01.763218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b7665cccd-249vg"
Sep 10 13:32:01.781002 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:01.780909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f"}
Sep 10 13:32:02.004065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.003964    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:32:02.307160 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.307125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:32:02.307360 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.307201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:32:02.307360 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.307236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:32:02.321848 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.321826    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 13:32:02.321952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.321929    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 13:32:02.322356 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.322342    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 13:32:02.330090 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.330065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a404e583-94f1-455b-b237-16adac92fe09-original-pull-secret\") pod \"global-pull-secret-syncer-695ls\" (UID: \"a404e583-94f1-455b-b237-16adac92fe09\") " pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:32:02.330178 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.330157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f81e5544-e11d-4c22-bbbc-e09f9d1034a9-metrics-certs\") pod \"network-metrics-daemon-txvf6\" (UID: \"f81e5544-e11d-4c22-bbbc-e09f9d1034a9\") " pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:32:02.340451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.340433    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 13:32:02.349924 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.349905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5srzq\" (UniqueName: \"kubernetes.io/projected/40df8a05-6418-4830-8b6a-0d810371127a-kube-api-access-5srzq\") pod \"network-check-target-2vrct\" (UID: \"40df8a05-6418-4830-8b6a-0d810371127a\") " pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:32:02.455597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.455521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-695ls"
Sep 10 13:32:02.462755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.462734    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twwrf\""
Sep 10 13:32:02.471108 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.471085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:32:02.563308 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.563285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nc4jc\""
Sep 10 13:32:02.570554 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.570507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txvf6"
Sep 10 13:32:02.754123 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.754038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-695ls"]
Sep 10 13:32:02.757263 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:32:02.757209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda404e583_94f1_455b_b237_16adac92fe09.slice/crio-a15be6353d380b245fe9bd50360505fd6737ceefb452b58fbbed3032e9eb2a41 WatchSource:0}: Error finding container a15be6353d380b245fe9bd50360505fd6737ceefb452b58fbbed3032e9eb2a41: Status 404 returned error can't find the container with id a15be6353d380b245fe9bd50360505fd6737ceefb452b58fbbed3032e9eb2a41
Sep 10 13:32:02.786540 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.786513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vrct"]
Sep 10 13:32:02.790461 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:32:02.790428    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40df8a05_6418_4830_8b6a_0d810371127a.slice/crio-4036cdd944048113c0148a45f03812952f6db1c3e2c8dfcc2f124b6c80b6e390 WatchSource:0}: Error finding container 4036cdd944048113c0148a45f03812952f6db1c3e2c8dfcc2f124b6c80b6e390: Status 404 returned error can't find the container with id 4036cdd944048113c0148a45f03812952f6db1c3e2c8dfcc2f124b6c80b6e390
Sep 10 13:32:02.790735 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.790704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf"}
Sep 10 13:32:02.790855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.790778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441"}
Sep 10 13:32:02.790855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.790832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c"}
Sep 10 13:32:02.790986 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.790876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e"}
Sep 10 13:32:02.790986 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.790916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerStarted","Data":"ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260"}
Sep 10 13:32:02.792538 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.792515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-695ls" event={"ID":"a404e583-94f1-455b-b237-16adac92fe09","Type":"ContainerStarted","Data":"a15be6353d380b245fe9bd50360505fd6737ceefb452b58fbbed3032e9eb2a41"}
Sep 10 13:32:02.833809 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.833758    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.922014918 podStartE2EDuration="8.833741378s" totalImagesPullingTime="4.885988252s" totalInitContainerRuntime="25.738208ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:02.831715724 +0000 UTC m=+66.856530431" watchObservedRunningTime="2026-09-10 13:32:02.833741378 +0000 UTC m=+66.858556129"
Sep 10 13:32:02.850989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:02.850960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-txvf6"]
Sep 10 13:32:02.853588 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:32:02.853551    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81e5544_e11d_4c22_bbbc_e09f9d1034a9.slice/crio-8f29056a7899d10c2e1b8a2a4a8ac5a393607b1c081ecdc43adfce762a39ff5b WatchSource:0}: Error finding container 8f29056a7899d10c2e1b8a2a4a8ac5a393607b1c081ecdc43adfce762a39ff5b: Status 404 returned error can't find the container with id 8f29056a7899d10c2e1b8a2a4a8ac5a393607b1c081ecdc43adfce762a39ff5b
Sep 10 13:32:03.617184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:03.617153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:03.644007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:03.643821    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:03.806644 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:03.806602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vrct" event={"ID":"40df8a05-6418-4830-8b6a-0d810371127a","Type":"ContainerStarted","Data":"4036cdd944048113c0148a45f03812952f6db1c3e2c8dfcc2f124b6c80b6e390"}
Sep 10 13:32:03.808564 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:03.808531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-txvf6" event={"ID":"f81e5544-e11d-4c22-bbbc-e09f9d1034a9","Type":"ContainerStarted","Data":"8f29056a7899d10c2e1b8a2a4a8ac5a393607b1c081ecdc43adfce762a39ff5b"}
Sep 10 13:32:03.809129 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:03.809109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:03.828749 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:03.828447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:04.816747 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:04.816614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-txvf6" event={"ID":"f81e5544-e11d-4c22-bbbc-e09f9d1034a9","Type":"ContainerStarted","Data":"a645e1feb4812056c7eb3b8c924a2b8a44187e2c8abeede9cfa62af1ac4a5d73"}
Sep 10 13:32:04.816747 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:04.816677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-txvf6" event={"ID":"f81e5544-e11d-4c22-bbbc-e09f9d1034a9","Type":"ContainerStarted","Data":"f76cc9bb8fffe09a1a9758cfde47ee4e5dccddca1cf4f6a06ee8535c4c3cb674"}
Sep 10 13:32:04.837496 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:04.837446    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-txvf6" podStartSLOduration=67.776310357 podStartE2EDuration="1m8.837426875s" totalImagesPullingTime="1.061116518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:04.836435645 +0000 UTC m=+68.861250349" watchObservedRunningTime="2026-09-10 13:32:04.837426875 +0000 UTC m=+68.862241579"
Sep 10 13:32:07.834017 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:07.833969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vrct" event={"ID":"40df8a05-6418-4830-8b6a-0d810371127a","Type":"ContainerStarted","Data":"a6b2dccd230ac3643b76b1c410899302a5a641c7b2ec033920d26fe406be62f9"}
Sep 10 13:32:07.834405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:07.834051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:32:07.835756 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:07.835728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-695ls" event={"ID":"a404e583-94f1-455b-b237-16adac92fe09","Type":"ContainerStarted","Data":"4c04d6f156945913ac155f270a1bd16a6287123c38e69903ac75fac1eae4aa9d"}
Sep 10 13:32:07.862840 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:07.862784    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2vrct" podStartSLOduration=67.25072915 podStartE2EDuration="1m11.862770769s" totalImagesPullingTime="4.612041619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:07.86190839 +0000 UTC m=+71.886723093" watchObservedRunningTime="2026-09-10 13:32:07.862770769 +0000 UTC m=+71.887585475"
Sep 10 13:32:07.886517 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:07.886468    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-695ls" podStartSLOduration=66.238441183 podStartE2EDuration="1m10.886453668s" totalImagesPullingTime="4.648012485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:07.886055676 +0000 UTC m=+71.910870378" watchObservedRunningTime="2026-09-10 13:32:07.886453668 +0000 UTC m=+71.911268369"
Sep 10 13:32:08.541695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:08.541659    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8545c46595-28ndv"]
Sep 10 13:32:13.537941 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:13.537903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:32:13.538308 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:13.537967    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:32:22.257811 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.257692    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7557cdb568-d4bnr" podUID="4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" containerName="console" containerID="cri-o://47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4" gracePeriod=15
Sep 10 13:32:22.515753 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.515694    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:32:22.685702 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-trusted-ca-bundle\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.685888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgmhn\" (UniqueName: \"kubernetes.io/projected/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-kube-api-access-sgmhn\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.685888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685747    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-oauth-config\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.685888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-config\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.685888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-oauth-serving-cert\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.685888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-service-ca\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.685888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.685881    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-serving-cert\") pod \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\" (UID: \"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d\") "
Sep 10 13:32:22.686226 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.686075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:22.686282 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.686249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-service-ca" (OuterVolumeSpecName: "service-ca") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:22.686338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.686265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-config" (OuterVolumeSpecName: "console-config") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:22.686535 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.686510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:22.688237 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.688204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-kube-api-access-sgmhn" (OuterVolumeSpecName: "kube-api-access-sgmhn") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "kube-api-access-sgmhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:32:22.688349 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.688313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:22.688391 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.688360    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" (UID: "4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:22.787315 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787264    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-oauth-serving-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.787315 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787283    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.787315 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787292    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-serving-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.787315 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787301    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-trusted-ca-bundle\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.787315 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmhn\" (UniqueName: \"kubernetes.io/projected/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-kube-api-access-sgmhn\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.787315 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787317    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-oauth-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.787538 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.787327    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d-console-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:22.898184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.898157    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" containerID="47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4" exitCode=0
Sep 10 13:32:22.898279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.898200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7557cdb568-d4bnr" event={"ID":"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d","Type":"ContainerDied","Data":"47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4"}
Sep 10 13:32:22.898279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.898207    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7557cdb568-d4bnr"
Sep 10 13:32:22.898279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.898245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7557cdb568-d4bnr" event={"ID":"4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d","Type":"ContainerDied","Data":"b7219e638c845fb191c53113898cec3c9f437ff468ff8c3f0f1a47d6085386bc"}
Sep 10 13:32:22.898279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.898266    2577 scope.go:117] "RemoveContainer" containerID="47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4"
Sep 10 13:32:22.906928 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.906910    2577 scope.go:117] "RemoveContainer" containerID="47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4"
Sep 10 13:32:22.907206 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:22.907187    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4\": container with ID starting with 47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4 not found: ID does not exist" containerID="47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4"
Sep 10 13:32:22.907264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.907213    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4"} err="failed to get container status \"47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4\": rpc error: code = NotFound desc = could not find container \"47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4\": container with ID starting with 47023c523690fd7730a1dd9b7aaf09582e9451bee660ff62c467a258a9279fe4 not found: ID does not exist"
Sep 10 13:32:22.919945 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.919920    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7557cdb568-d4bnr"]
Sep 10 13:32:22.931779 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:22.931756    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7557cdb568-d4bnr"]
Sep 10 13:32:24.451083 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:24.451052    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" path="/var/lib/kubelet/pods/4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d/volumes"
Sep 10 13:32:33.543310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.543279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:32:33.547056 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.547037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fc48648d6-rpkvd"
Sep 10 13:32:33.566295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.566264    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8545c46595-28ndv" podUID="cc661eb4-1600-4c53-9adb-c6ea79c73b1a" containerName="console" containerID="cri-o://930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf" gracePeriod=15
Sep 10 13:32:33.800709 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.800651    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:32:33.869480 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-oauth-serving-cert\") pod \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") "
Sep 10 13:32:33.869650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr77t\" (UniqueName: \"kubernetes.io/projected/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-kube-api-access-cr77t\") pod \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") "
Sep 10 13:32:33.869650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-serving-cert\") pod \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") "
Sep 10 13:32:33.869650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-service-ca\") pod \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") "
Sep 10 13:32:33.869650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-config\") pod \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") "
Sep 10 13:32:33.869650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-oauth-config\") pod \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\" (UID: \"cc661eb4-1600-4c53-9adb-c6ea79c73b1a\") "
Sep 10 13:32:33.869944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.869915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cc661eb4-1600-4c53-9adb-c6ea79c73b1a" (UID: "cc661eb4-1600-4c53-9adb-c6ea79c73b1a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:33.870025 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.870001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-service-ca" (OuterVolumeSpecName: "service-ca") pod "cc661eb4-1600-4c53-9adb-c6ea79c73b1a" (UID: "cc661eb4-1600-4c53-9adb-c6ea79c73b1a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:33.870071 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.870011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-config" (OuterVolumeSpecName: "console-config") pod "cc661eb4-1600-4c53-9adb-c6ea79c73b1a" (UID: "cc661eb4-1600-4c53-9adb-c6ea79c73b1a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:33.871736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.871711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-kube-api-access-cr77t" (OuterVolumeSpecName: "kube-api-access-cr77t") pod "cc661eb4-1600-4c53-9adb-c6ea79c73b1a" (UID: "cc661eb4-1600-4c53-9adb-c6ea79c73b1a"). InnerVolumeSpecName "kube-api-access-cr77t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:32:33.871736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.871732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cc661eb4-1600-4c53-9adb-c6ea79c73b1a" (UID: "cc661eb4-1600-4c53-9adb-c6ea79c73b1a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:33.871866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.871772    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cc661eb4-1600-4c53-9adb-c6ea79c73b1a" (UID: "cc661eb4-1600-4c53-9adb-c6ea79c73b1a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:33.944230 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.944206    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc661eb4-1600-4c53-9adb-c6ea79c73b1a" containerID="930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf" exitCode=0
Sep 10 13:32:33.944338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.944256    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8545c46595-28ndv"
Sep 10 13:32:33.944338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.944294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8545c46595-28ndv" event={"ID":"cc661eb4-1600-4c53-9adb-c6ea79c73b1a","Type":"ContainerDied","Data":"930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf"}
Sep 10 13:32:33.944418 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.944343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8545c46595-28ndv" event={"ID":"cc661eb4-1600-4c53-9adb-c6ea79c73b1a","Type":"ContainerDied","Data":"fc708e2a075f8dc103e8524bd6757cf9b1774cc8aadad53238ee08310c0d791b"}
Sep 10 13:32:33.944418 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.944363    2577 scope.go:117] "RemoveContainer" containerID="930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf"
Sep 10 13:32:33.952480 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.952464    2577 scope.go:117] "RemoveContainer" containerID="930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf"
Sep 10 13:32:33.952838 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:33.952802    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf\": container with ID starting with 930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf not found: ID does not exist" containerID="930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf"
Sep 10 13:32:33.952928 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.952842    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf"} err="failed to get container status \"930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf\": rpc error: code = NotFound desc = could not find container \"930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf\": container with ID starting with 930b89ac1286ee82f8783be04e7bebeb1c6b7772c38ecca2d0a81c9d65c89bcf not found: ID does not exist"
Sep 10 13:32:33.966697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.966675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8545c46595-28ndv"]
Sep 10 13:32:33.970422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.970401    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-oauth-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:33.970523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.970425    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-oauth-serving-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:33.970523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.970440    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr77t\" (UniqueName: \"kubernetes.io/projected/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-kube-api-access-cr77t\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:33.970523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.970457    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-serving-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:33.970523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.970470    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:33.970523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.970484    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc661eb4-1600-4c53-9adb-c6ea79c73b1a-console-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:33.975281 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:33.975259    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8545c46595-28ndv"]
Sep 10 13:32:34.452206 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:34.451868    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc661eb4-1600-4c53-9adb-c6ea79c73b1a" path="/var/lib/kubelet/pods/cc661eb4-1600-4c53-9adb-c6ea79c73b1a/volumes"
Sep 10 13:32:38.596692 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.596654    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:38.597229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.596936    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="alertmanager" containerID="cri-o://713092130a9bef5d4f4bcad9b219d2ca1cec08c6d09b3d1b3ceb0998923e5d5e" gracePeriod=120
Sep 10 13:32:38.597229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.596993    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-metric" containerID="cri-o://03900e71c765cb99775032b549a545d3d09db30f4ba14806ae709964c2c83d21" gracePeriod=120
Sep 10 13:32:38.597229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.597059    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="prom-label-proxy" containerID="cri-o://bafda016d2f414108f2bacabecd5303ad88df2dbb311fb9d46769e5fe7dd4d40" gracePeriod=120
Sep 10 13:32:38.597229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.597037    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-web" containerID="cri-o://eeb1954bc2875a362cc61375dad3f7062219464c2073402fb2259da4f10feee7" gracePeriod=120
Sep 10 13:32:38.597229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.597105    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="config-reloader" containerID="cri-o://05939605d06d386ced7b75bcb1fa86722b1060b8d740c42e95611839f2a41915" gracePeriod=120
Sep 10 13:32:38.597229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.597081    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy" containerID="cri-o://e9ef98f9e53e0aa0cc927577b032a35d3d7e0e089ac71c6acbd702ee6cc18ed4" gracePeriod=120
Sep 10 13:32:38.842333 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.842309    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2vrct"
Sep 10 13:32:38.974738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974661    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="bafda016d2f414108f2bacabecd5303ad88df2dbb311fb9d46769e5fe7dd4d40" exitCode=0
Sep 10 13:32:38.974738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974691    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="e9ef98f9e53e0aa0cc927577b032a35d3d7e0e089ac71c6acbd702ee6cc18ed4" exitCode=0
Sep 10 13:32:38.974738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974698    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="05939605d06d386ced7b75bcb1fa86722b1060b8d740c42e95611839f2a41915" exitCode=0
Sep 10 13:32:38.974738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974705    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="713092130a9bef5d4f4bcad9b219d2ca1cec08c6d09b3d1b3ceb0998923e5d5e" exitCode=0
Sep 10 13:32:38.974738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"bafda016d2f414108f2bacabecd5303ad88df2dbb311fb9d46769e5fe7dd4d40"}
Sep 10 13:32:38.974976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"e9ef98f9e53e0aa0cc927577b032a35d3d7e0e089ac71c6acbd702ee6cc18ed4"}
Sep 10 13:32:38.974976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"05939605d06d386ced7b75bcb1fa86722b1060b8d740c42e95611839f2a41915"}
Sep 10 13:32:38.974976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:38.974862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"713092130a9bef5d4f4bcad9b219d2ca1cec08c6d09b3d1b3ceb0998923e5d5e"}
Sep 10 13:32:39.982693 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:39.982616    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="03900e71c765cb99775032b549a545d3d09db30f4ba14806ae709964c2c83d21" exitCode=0
Sep 10 13:32:39.982693 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:39.982641    2577 generic.go:356] "Generic (PLEG): container finished" podID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerID="eeb1954bc2875a362cc61375dad3f7062219464c2073402fb2259da4f10feee7" exitCode=0
Sep 10 13:32:39.982693 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:39.982685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"03900e71c765cb99775032b549a545d3d09db30f4ba14806ae709964c2c83d21"}
Sep 10 13:32:39.983134 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:39.982742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"eeb1954bc2875a362cc61375dad3f7062219464c2073402fb2259da4f10feee7"}
Sep 10 13:32:40.379105 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.379080    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:40.423350 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-main-tls\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-cluster-tls-config\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-main-db\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-metrics-client-ca\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-tls-assets\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkxkz\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-kube-api-access-fkxkz\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423749 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-volume\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423865 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:32:40.423865 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:40.423968 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423874    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-out\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423968 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423920    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-web-config\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.423968 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-web\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.424113 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.423989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.424113 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.424016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.424113 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.424087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"83c3e51c-3958-47e2-9063-c4a084fceaa0\" (UID: \"83c3e51c-3958-47e2-9063-c4a084fceaa0\") "
Sep 10 13:32:40.425192 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.424345    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-main-db\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.425192 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.424368    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-metrics-client-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.425192 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.424901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:40.426957 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.426893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-volume" (OuterVolumeSpecName: "config-volume") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.427616 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.427510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-out" (OuterVolumeSpecName: "config-out") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:32:40.427616 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.427592    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.428566 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.428522    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-kube-api-access-fkxkz" (OuterVolumeSpecName: "kube-api-access-fkxkz") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "kube-api-access-fkxkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:32:40.428776 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.428744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:32:40.429308 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.429260    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.429570 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.429479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.429733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.429707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.434474 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.434443    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.440848 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.440822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-web-config" (OuterVolumeSpecName: "web-config") pod "83c3e51c-3958-47e2-9063-c4a084fceaa0" (UID: "83c3e51c-3958-47e2-9063-c4a084fceaa0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:40.525614 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525521    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525614 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525548    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-main-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525614 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525561    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-cluster-tls-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525614 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525573    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-tls-assets\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525614 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525605    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxkz\" (UniqueName: \"kubernetes.io/projected/83c3e51c-3958-47e2-9063-c4a084fceaa0-kube-api-access-fkxkz\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525620    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525632    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c3e51c-3958-47e2-9063-c4a084fceaa0-config-out\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525643    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-web-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525654    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525667    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c3e51c-3958-47e2-9063-c4a084fceaa0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.525857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.525679    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c3e51c-3958-47e2-9063-c4a084fceaa0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:40.989431 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.989394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"83c3e51c-3958-47e2-9063-c4a084fceaa0","Type":"ContainerDied","Data":"e0e7bbebcbba603c78ba8db9abfcd84129149e19a37e7223361dedbe458ef69b"}
Sep 10 13:32:40.989849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.989437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:40.989849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.989473    2577 scope.go:117] "RemoveContainer" containerID="bafda016d2f414108f2bacabecd5303ad88df2dbb311fb9d46769e5fe7dd4d40"
Sep 10 13:32:40.997807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:40.997792    2577 scope.go:117] "RemoveContainer" containerID="03900e71c765cb99775032b549a545d3d09db30f4ba14806ae709964c2c83d21"
Sep 10 13:32:41.004765 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.004749    2577 scope.go:117] "RemoveContainer" containerID="e9ef98f9e53e0aa0cc927577b032a35d3d7e0e089ac71c6acbd702ee6cc18ed4"
Sep 10 13:32:41.011133 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.011118    2577 scope.go:117] "RemoveContainer" containerID="eeb1954bc2875a362cc61375dad3f7062219464c2073402fb2259da4f10feee7"
Sep 10 13:32:41.017094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.017045    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:41.019327 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.019310    2577 scope.go:117] "RemoveContainer" containerID="05939605d06d386ced7b75bcb1fa86722b1060b8d740c42e95611839f2a41915"
Sep 10 13:32:41.041973 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.041921    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:41.049342 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.049321    2577 scope.go:117] "RemoveContainer" containerID="713092130a9bef5d4f4bcad9b219d2ca1cec08c6d09b3d1b3ceb0998923e5d5e"
Sep 10 13:32:41.057192 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.057173    2577 scope.go:117] "RemoveContainer" containerID="5d7bcb514e45f155aeaef1c7ec3720abffcc37bb3bc656f2a008160cfcc8b45e"
Sep 10 13:32:41.066134 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066111    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:41.066454 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066439    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="init-config-reloader"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066456    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="init-config-reloader"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="alertmanager"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066480    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="alertmanager"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066491    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc661eb4-1600-4c53-9adb-c6ea79c73b1a" containerName="console"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066500    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc661eb4-1600-4c53-9adb-c6ea79c73b1a" containerName="console"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066509    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" containerName="console"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066517    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" containerName="console"
Sep 10 13:32:41.066527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066529    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-metric"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066537    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-metric"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="config-reloader"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="config-reloader"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066570    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-web"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066593    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-web"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066616    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="prom-label-proxy"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066625    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="prom-label-proxy"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066647    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066729    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ec7c1d0-86fa-4915-a7b7-ed5b4925b12d" containerName="console"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066743    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-metric"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066752    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="prom-label-proxy"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066766    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy-web"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066775    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc661eb4-1600-4c53-9adb-c6ea79c73b1a" containerName="console"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066785    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="alertmanager"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066794    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="config-reloader"
Sep 10 13:32:41.066933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.066804    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" containerName="kube-rbac-proxy"
Sep 10 13:32:41.099961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.099941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:41.100093 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.100079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.102677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.102650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 13:32:41.103046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103014    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 13:32:41.103171 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103156    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 13:32:41.103251 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 13:32:41.103348 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 13:32:41.103466 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103451    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tknhq\""
Sep 10 13:32:41.103706 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 13:32:41.103706 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103688    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 13:32:41.103928 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.103793    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 13:32:41.108467 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.108447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 13:32:41.238545 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238545 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238776 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238776 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238776 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238776 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-web-config\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.238989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.238974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-config-out\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.239151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.239001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xm2t\" (UniqueName: \"kubernetes.io/projected/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-kube-api-access-8xm2t\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.239151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.239027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.239151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.239049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340143 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340281 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340281 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-web-config\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-config-out\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340860 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xm2t\" (UniqueName: \"kubernetes.io/projected/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-kube-api-access-8xm2t\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.340860 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.340627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.341154 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.341129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.342237 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.342212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343153 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343258 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343361 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343489 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343654 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-config-out\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343756 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.343817 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.343798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.345119 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.345091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-web-config\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.345188 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.345163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.352824 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.352804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xm2t\" (UniqueName: \"kubernetes.io/projected/ca4f0559-e610-4b7a-ad16-2b7a4df308bf-kube-api-access-8xm2t\") pod \"alertmanager-main-0\" (UID: \"ca4f0559-e610-4b7a-ad16-2b7a4df308bf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.410755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.410732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:41.742058 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.741979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:41.745368 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:32:41.745343    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4f0559_e610_4b7a_ad16_2b7a4df308bf.slice/crio-bb0929fb8d540d25a67d706bad8ed6d882a4584ba96cb474508c5ef90e5938b4 WatchSource:0}: Error finding container bb0929fb8d540d25a67d706bad8ed6d882a4584ba96cb474508c5ef90e5938b4: Status 404 returned error can't find the container with id bb0929fb8d540d25a67d706bad8ed6d882a4584ba96cb474508c5ef90e5938b4
Sep 10 13:32:41.999218 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.999137    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca4f0559-e610-4b7a-ad16-2b7a4df308bf" containerID="b14e322a43f9712f3ff7ba9240a9e33145858418524cf0a15cc5493cb5fde82d" exitCode=0
Sep 10 13:32:41.999651 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.999221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerDied","Data":"b14e322a43f9712f3ff7ba9240a9e33145858418524cf0a15cc5493cb5fde82d"}
Sep 10 13:32:41.999651 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:41.999310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"bb0929fb8d540d25a67d706bad8ed6d882a4584ba96cb474508c5ef90e5938b4"}
Sep 10 13:32:42.450652 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.450619    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c3e51c-3958-47e2-9063-c4a084fceaa0" path="/var/lib/kubelet/pods/83c3e51c-3958-47e2-9063-c4a084fceaa0/volumes"
Sep 10 13:32:42.682801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.681482    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-95d898cb-kb8mp"]
Sep 10 13:32:42.685416 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.685395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.688032 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.687988    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 13:32:42.688032 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.688002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 13:32:42.688032 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.688000    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 13:32:42.688283 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.688127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xztsc\""
Sep 10 13:32:42.688722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.688699    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 13:32:42.688722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.688722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 13:32:42.693391 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.693354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 13:32:42.701574 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.701545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-95d898cb-kb8mp"]
Sep 10 13:32:42.754082 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-telemeter-client-tls\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-serving-certs-ca-bundle\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-federate-client-tls\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5n4\" (UniqueName: \"kubernetes.io/projected/20044fa4-cf73-436a-ad78-a762e91f3d05-kube-api-access-rd5n4\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-metrics-client-ca\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-secret-telemeter-client\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.754437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.754369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-telemeter-trusted-ca-bundle\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.855895 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.855850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-telemeter-client-tls\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.856095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.855926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-serving-certs-ca-bundle\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.856095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.855964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-federate-client-tls\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.856095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.855986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5n4\" (UniqueName: \"kubernetes.io/projected/20044fa4-cf73-436a-ad78-a762e91f3d05-kube-api-access-rd5n4\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.856095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-metrics-client-ca\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.856095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.857352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-secret-telemeter-client\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.857352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-telemeter-trusted-ca-bundle\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.857352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-serving-certs-ca-bundle\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.857352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-metrics-client-ca\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.857352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.856988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20044fa4-cf73-436a-ad78-a762e91f3d05-telemeter-trusted-ca-bundle\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.859551 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.859183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-telemeter-client-tls\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.859878 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.859857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-federate-client-tls\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.859931 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.859909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.860563 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.860540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/20044fa4-cf73-436a-ad78-a762e91f3d05-secret-telemeter-client\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.868910 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.868883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5n4\" (UniqueName: \"kubernetes.io/projected/20044fa4-cf73-436a-ad78-a762e91f3d05-kube-api-access-rd5n4\") pod \"telemeter-client-95d898cb-kb8mp\" (UID: \"20044fa4-cf73-436a-ad78-a762e91f3d05\") " pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:42.990548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.990469    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:42.990892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.990857    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="prometheus" containerID="cri-o://3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f" gracePeriod=600
Sep 10 13:32:42.991761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.991117    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy" containerID="cri-o://6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441" gracePeriod=600
Sep 10 13:32:42.991761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.991220    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-web" containerID="cri-o://2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c" gracePeriod=600
Sep 10 13:32:42.991761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.991319    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="thanos-sidecar" containerID="cri-o://f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e" gracePeriod=600
Sep 10 13:32:42.991761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.991390    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="config-reloader" containerID="cri-o://ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260" gracePeriod=600
Sep 10 13:32:42.991761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.991618    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf" gracePeriod=600
Sep 10 13:32:42.995963 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:42.995732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp"
Sep 10 13:32:43.013438 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.013406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"de1225c2463117676488824170e8e4231fa487fa48ca1bfff3583f09400af191"}
Sep 10 13:32:43.013875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.013504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"9f7f6b2fbf51767ce6f112e10cd088b611e415736c9463551c688ed36fa2910e"}
Sep 10 13:32:43.013875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.013553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"06f19624f51f48c4b76da2f1a6834c7f6bcb8766ce7c5c672d1a9037ef8b7aad"}
Sep 10 13:32:43.013875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.013608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"bc1dd367ca2034dd66bcf23d1196fbdf232640fdcff38c767f39581c231820a4"}
Sep 10 13:32:43.013875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.013663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"dc684948851d3211b5fa7ca1d0bef43e79d561b6e76c073008dbd6093ae0bd15"}
Sep 10 13:32:43.013875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.013706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca4f0559-e610-4b7a-ad16-2b7a4df308bf","Type":"ContainerStarted","Data":"525261b20dcd8296701378cba2221643d39202af919fc9d7aee8873a18bfc989"}
Sep 10 13:32:43.075049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.074967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.059281551 podStartE2EDuration="2.074930868s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.649317ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:43.06939277 +0000 UTC m=+107.094207535" watchObservedRunningTime="2026-09-10 13:32:43.074930868 +0000 UTC m=+107.099745574"
Sep 10 13:32:43.312378 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:43.312193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-95d898cb-kb8mp"]
Sep 10 13:32:43.314527 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:32:43.314492    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20044fa4_cf73_436a_ad78_a762e91f3d05.slice/crio-19d9628a682bfc27ba3e05b77c7255a5005b97e5028156c17e88bfd43a70643f WatchSource:0}: Error finding container 19d9628a682bfc27ba3e05b77c7255a5005b97e5028156c17e88bfd43a70643f: Status 404 returned error can't find the container with id 19d9628a682bfc27ba3e05b77c7255a5005b97e5028156c17e88bfd43a70643f
Sep 10 13:32:43.809679 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:43.809624    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f is running failed: container process not found" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 10 13:32:43.810051 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:43.810010    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f is running failed: container process not found" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 10 13:32:43.810513 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:43.810385    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f is running failed: container process not found" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 10 13:32:43.810513 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:43.810442    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f is running failed: container process not found" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="prometheus" probeResult="unknown"
Sep 10 13:32:44.027660 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027562    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf" exitCode=0
Sep 10 13:32:44.027660 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027605    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441" exitCode=0
Sep 10 13:32:44.027660 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027615    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e" exitCode=0
Sep 10 13:32:44.027660 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf"}
Sep 10 13:32:44.028170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441"}
Sep 10 13:32:44.028170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027622    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260" exitCode=0
Sep 10 13:32:44.028170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e"}
Sep 10 13:32:44.028170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027733    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f" exitCode=0
Sep 10 13:32:44.028170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260"}
Sep 10 13:32:44.028170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.027784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f"}
Sep 10 13:32:44.029429 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.029407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp" event={"ID":"20044fa4-cf73-436a-ad78-a762e91f3d05","Type":"ContainerStarted","Data":"19d9628a682bfc27ba3e05b77c7255a5005b97e5028156c17e88bfd43a70643f"}
Sep 10 13:32:44.289219 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.289147    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.372909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-serving-certs-ca-bundle\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.372960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-1\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373001    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-grpc-tls\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-db\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-tls\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373098    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-config\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373130    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-kubelet-serving-ca-bundle\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmdxv\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-kube-api-access-jmdxv\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373261    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-config-out\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-web-config\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-metrics-client-ca\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-2\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-0\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373524    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-thanos-prometheus-http-client-file\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-kube-rbac-proxy\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.374607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.373991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-tls-assets\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.375726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.374034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-trusted-ca-bundle\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.375726 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.374076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-metrics-client-certs\") pod \"43015a99-800c-4032-92b0-9efdb8cecb8b\" (UID: \"43015a99-800c-4032-92b0-9efdb8cecb8b\") "
Sep 10 13:32:44.376257 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.376225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.376861 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.376829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.377875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.377774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.380401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.378200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.380401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.379219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.380401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.379888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.380401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.380076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-kube-api-access-jmdxv" (OuterVolumeSpecName: "kube-api-access-jmdxv") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "kube-api-access-jmdxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:32:44.381827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.381491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:32:44.381827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.381784    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:32:44.383056 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.383026    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.383215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.383191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-config-out" (OuterVolumeSpecName: "config-out") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:32:44.383492 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.383453    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.384205 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.384165    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.384494 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.384458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-config" (OuterVolumeSpecName: "config") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.384990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.384941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.385314 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.385276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.385400 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.385303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:32:44.385918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.385878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.387218 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.387178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.394775 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.394748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-web-config" (OuterVolumeSpecName: "web-config") pod "43015a99-800c-4032-92b0-9efdb8cecb8b" (UID: "43015a99-800c-4032-92b0-9efdb8cecb8b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:32:44.475041 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475000    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdxv\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-kube-api-access-jmdxv\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475041 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475034    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-config-out\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475041 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475047    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-web-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475059    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475073    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-metrics-client-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475085    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475098    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475110    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475122    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-kube-rbac-proxy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475133    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43015a99-800c-4032-92b0-9efdb8cecb8b-tls-assets\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475145    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475158    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-metrics-client-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475170    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475182    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475194    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-grpc-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475206    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43015a99-800c-4032-92b0-9efdb8cecb8b-prometheus-k8s-db\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475220    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475234    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-config\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475247    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43015a99-800c-4032-92b0-9efdb8cecb8b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:44.475299 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:44.475261    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43015a99-800c-4032-92b0-9efdb8cecb8b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:32:45.038340 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.038267    2577 generic.go:356] "Generic (PLEG): container finished" podID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerID="2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c" exitCode=0
Sep 10 13:32:45.038340 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.038318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c"}
Sep 10 13:32:45.038759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.038398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43015a99-800c-4032-92b0-9efdb8cecb8b","Type":"ContainerDied","Data":"0cd60b62b9ebd8951048face164649542803d39fd9ee77d8d7b38e95e81e1c88"}
Sep 10 13:32:45.038759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.038422    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.038759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.038447    2577 scope.go:117] "RemoveContainer" containerID="7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf"
Sep 10 13:32:45.066624 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.066595    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:45.075388 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.075364    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:45.107672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.107642    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:45.108019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.107997    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="init-config-reloader"
Sep 10 13:32:45.108019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="init-config-reloader"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108027    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="thanos-sidecar"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108036    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="thanos-sidecar"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108048    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-thanos"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108057    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-thanos"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108079    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-web"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108087    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-web"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108097    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="prometheus"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108105    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="prometheus"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108113    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="config-reloader"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108121    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="config-reloader"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108133    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy"
Sep 10 13:32:45.108197 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108141    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy"
Sep 10 13:32:45.108772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108229    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="prometheus"
Sep 10 13:32:45.108772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108243    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-web"
Sep 10 13:32:45.108772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108252    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy"
Sep 10 13:32:45.108772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108259    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="kube-rbac-proxy-thanos"
Sep 10 13:32:45.108772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108270    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="config-reloader"
Sep 10 13:32:45.108772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.108280    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" containerName="thanos-sidecar"
Sep 10 13:32:45.111834 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.111811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.114687 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.114665    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 13:32:45.115519 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.115498    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 13:32:45.115754 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.115731    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7vwfr\""
Sep 10 13:32:45.115930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.115914    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116136    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116231    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116240    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116359    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f43cgjb55jclu\""
Sep 10 13:32:45.116405 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.116380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 13:32:45.117092 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.117065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 13:32:45.118979 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.118840    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 13:32:45.122262 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.121245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 13:32:45.141591 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.141558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:45.181785 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.181723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.181785 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.181782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.181861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.181922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6tw\" (UniqueName: \"kubernetes.io/projected/1857d2fc-d81f-4bac-a30d-a839e3974be7-kube-api-access-bz6tw\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.181956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.181986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1857d2fc-d81f-4bac-a30d-a839e3974be7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182432 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182432 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-config\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182432 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1857d2fc-d81f-4bac-a30d-a839e3974be7-config-out\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182432 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182432 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-web-config\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182432 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.182733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.182507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283702 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-config\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1857d2fc-d81f-4bac-a30d-a839e3974be7-config-out\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.283892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-web-config\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.283970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6tw\" (UniqueName: \"kubernetes.io/projected/1857d2fc-d81f-4bac-a30d-a839e3974be7-kube-api-access-bz6tw\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1857d2fc-d81f-4bac-a30d-a839e3974be7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.284861 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.284700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.285331 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.285304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.287400 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.287223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.287400 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.287373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.287712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1857d2fc-d81f-4bac-a30d-a839e3974be7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.287767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-config\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.288111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.288380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.288476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.288767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.288805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.288789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.289514 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.289461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.289689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.289653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1857d2fc-d81f-4bac-a30d-a839e3974be7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.289934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.289912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-web-config\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.290040 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.289930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1857d2fc-d81f-4bac-a30d-a839e3974be7-config-out\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.290477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.290414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.290845 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.290807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.291555 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.291534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.291672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.291538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1857d2fc-d81f-4bac-a30d-a839e3974be7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.298159 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.298130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6tw\" (UniqueName: \"kubernetes.io/projected/1857d2fc-d81f-4bac-a30d-a839e3974be7-kube-api-access-bz6tw\") pod \"prometheus-k8s-0\" (UID: \"1857d2fc-d81f-4bac-a30d-a839e3974be7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.426286 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.426244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:45.437242 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.437148    2577 scope.go:117] "RemoveContainer" containerID="6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441"
Sep 10 13:32:45.478556 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.478512    2577 scope.go:117] "RemoveContainer" containerID="2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c"
Sep 10 13:32:45.487051 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.487015    2577 scope.go:117] "RemoveContainer" containerID="f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e"
Sep 10 13:32:45.496763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.496741    2577 scope.go:117] "RemoveContainer" containerID="ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260"
Sep 10 13:32:45.523370 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.512142    2577 scope.go:117] "RemoveContainer" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f"
Sep 10 13:32:45.541760 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.541724    2577 scope.go:117] "RemoveContainer" containerID="062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99"
Sep 10 13:32:45.553074 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.553045    2577 scope.go:117] "RemoveContainer" containerID="7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf"
Sep 10 13:32:45.553457 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.553434    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf\": container with ID starting with 7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf not found: ID does not exist" containerID="7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf"
Sep 10 13:32:45.553526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.553467    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf"} err="failed to get container status \"7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf\": rpc error: code = NotFound desc = could not find container \"7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf\": container with ID starting with 7759cb4ea67014d3dea10682bdf1ceff0fe6c025304d1a8be729e7a20e3a2dcf not found: ID does not exist"
Sep 10 13:32:45.553526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.553488    2577 scope.go:117] "RemoveContainer" containerID="6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441"
Sep 10 13:32:45.553824 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.553807    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441\": container with ID starting with 6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441 not found: ID does not exist" containerID="6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441"
Sep 10 13:32:45.553872 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.553828    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441"} err="failed to get container status \"6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441\": rpc error: code = NotFound desc = could not find container \"6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441\": container with ID starting with 6eb769072a761ef2469a0f73ef795f020d2d7e2bbbedd976b1623da0c5c6a441 not found: ID does not exist"
Sep 10 13:32:45.553872 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.553842    2577 scope.go:117] "RemoveContainer" containerID="2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c"
Sep 10 13:32:45.554128 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.554100    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c\": container with ID starting with 2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c not found: ID does not exist" containerID="2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c"
Sep 10 13:32:45.554196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.554136    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c"} err="failed to get container status \"2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c\": rpc error: code = NotFound desc = could not find container \"2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c\": container with ID starting with 2d94b16e09376c957ca0e6837ee76912ae0ed6b71b7ca3fc7682bd0f74862c8c not found: ID does not exist"
Sep 10 13:32:45.554196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.554164    2577 scope.go:117] "RemoveContainer" containerID="f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e"
Sep 10 13:32:45.554469 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.554447    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e\": container with ID starting with f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e not found: ID does not exist" containerID="f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e"
Sep 10 13:32:45.554549 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.554476    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e"} err="failed to get container status \"f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e\": rpc error: code = NotFound desc = could not find container \"f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e\": container with ID starting with f43e8d5949f321afd014bf2d798d8ddbc2d9ca9bcc263774328ba84932b25a8e not found: ID does not exist"
Sep 10 13:32:45.554549 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.554495    2577 scope.go:117] "RemoveContainer" containerID="ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260"
Sep 10 13:32:45.554996 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.554919    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260\": container with ID starting with ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260 not found: ID does not exist" containerID="ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260"
Sep 10 13:32:45.554996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.554945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260"} err="failed to get container status \"ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260\": rpc error: code = NotFound desc = could not find container \"ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260\": container with ID starting with ddfc39f5602a1b4c21928e97694eb6a7ffd8d1e75cf72db8c044906b327a1260 not found: ID does not exist"
Sep 10 13:32:45.554996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.554966    2577 scope.go:117] "RemoveContainer" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f"
Sep 10 13:32:45.555396 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.555377    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f\": container with ID starting with 3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f not found: ID does not exist" containerID="3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f"
Sep 10 13:32:45.555457 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.555401    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f"} err="failed to get container status \"3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f\": rpc error: code = NotFound desc = could not find container \"3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f\": container with ID starting with 3b51acb5da2f16e6580703142118da80fdc412a05c4d6b1a9affb73869708f0f not found: ID does not exist"
Sep 10 13:32:45.555457 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.555420    2577 scope.go:117] "RemoveContainer" containerID="062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99"
Sep 10 13:32:45.555729 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:32:45.555698    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99\": container with ID starting with 062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99 not found: ID does not exist" containerID="062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99"
Sep 10 13:32:45.555816 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.555723    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99"} err="failed to get container status \"062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99\": rpc error: code = NotFound desc = could not find container \"062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99\": container with ID starting with 062af4efb49650a29f43cc3281c07b4a88ec2c26a895c0801584bb93041faa99 not found: ID does not exist"
Sep 10 13:32:45.761281 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:45.759885    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:45.775762 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:32:45.775728    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1857d2fc_d81f_4bac_a30d_a839e3974be7.slice/crio-a781e468961bc34239bef2dd360b683726511f3d23fc4c980d97e7bb15886e23 WatchSource:0}: Error finding container a781e468961bc34239bef2dd360b683726511f3d23fc4c980d97e7bb15886e23: Status 404 returned error can't find the container with id a781e468961bc34239bef2dd360b683726511f3d23fc4c980d97e7bb15886e23
Sep 10 13:32:46.044801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.044758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp" event={"ID":"20044fa4-cf73-436a-ad78-a762e91f3d05","Type":"ContainerStarted","Data":"5cd2746b499b06d2296ea27ce0d83b9597e99bd6ec9a5a4c6ca72929631a358a"}
Sep 10 13:32:46.045280 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.044827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp" event={"ID":"20044fa4-cf73-436a-ad78-a762e91f3d05","Type":"ContainerStarted","Data":"30b1d1c223e50d18c7a035187091176c18589c700e44e03afcdd5b764dbeffb3"}
Sep 10 13:32:46.045280 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.044855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp" event={"ID":"20044fa4-cf73-436a-ad78-a762e91f3d05","Type":"ContainerStarted","Data":"dfde47c5616628d270455bc26a19ef896875af03d0fcd33a72dd1e3433423930"}
Sep 10 13:32:46.046710 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.046685    2577 generic.go:356] "Generic (PLEG): container finished" podID="1857d2fc-d81f-4bac-a30d-a839e3974be7" containerID="8cade32ca82e6635b724a09df24acf431e6571d9d40528253f2e199c7483a83a" exitCode=0
Sep 10 13:32:46.046823 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.046782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerDied","Data":"8cade32ca82e6635b724a09df24acf431e6571d9d40528253f2e199c7483a83a"}
Sep 10 13:32:46.046863 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.046847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"a781e468961bc34239bef2dd360b683726511f3d23fc4c980d97e7bb15886e23"}
Sep 10 13:32:46.077161 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.077099    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-95d898cb-kb8mp" podStartSLOduration=1.911043423 podStartE2EDuration="4.077074278s" totalImagesPullingTime="2.166030855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:46.074709391 +0000 UTC m=+110.099524119" watchObservedRunningTime="2026-09-10 13:32:46.077074278 +0000 UTC m=+110.101889006"
Sep 10 13:32:46.457019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:46.455974    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43015a99-800c-4032-92b0-9efdb8cecb8b" path="/var/lib/kubelet/pods/43015a99-800c-4032-92b0-9efdb8cecb8b/volumes"
Sep 10 13:32:47.061505 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.061411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"7b830e359de6d0c9cbfb82920f665e92ef9d6a7b2bc219a234256cb217f4170c"}
Sep 10 13:32:47.061927 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.061530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"b39acca221e578e0fcb8ca8ea1455a71f727ee60be15e6c590d35d4463153381"}
Sep 10 13:32:47.061927 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.061592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"2183b07400764073b92880eca81072e8541190e69c50a326a459e00ed6404447"}
Sep 10 13:32:47.061927 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.061636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"a5ff096f91d1ca80f89847cb43f4e97c871f56d9a5315ab0f4c6a1be1596ae9b"}
Sep 10 13:32:47.061927 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.061674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"b51d311d591b7f3f90279317161aeaf4b59ace049227b69a5123d1a4c3e71edc"}
Sep 10 13:32:47.061927 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.061714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1857d2fc-d81f-4bac-a30d-a839e3974be7","Type":"ContainerStarted","Data":"92b8a1cef9dc8308e7e90b8483a7e7f49679c0ba642e68f14e3d64e3ff6f5c4b"}
Sep 10 13:32:47.148728 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:47.148653    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.115028843 podStartE2EDuration="2.148631059s" totalImagesPullingTime="0s" totalInitContainerRuntime="33.602216ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:47.14606615 +0000 UTC m=+111.170880854" watchObservedRunningTime="2026-09-10 13:32:47.148631059 +0000 UTC m=+111.173445763"
Sep 10 13:32:50.427216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:32:50.427171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:33:45.427244 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:33:45.427182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:33:45.444617 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:33:45.444567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:33:46.311447 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:33:46.311421    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:35:30.451124 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.451051    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-857qn"]
Sep 10 13:35:30.453378 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.453354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.457554 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.457531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 13:35:30.458212 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.458196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tffjg\""
Sep 10 13:35:30.458317 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.458213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 13:35:30.462907 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.462886    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-857qn"]
Sep 10 13:35:30.561505 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.561465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f9b7f2-8cc7-48eb-9906-c9bea914073a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-857qn\" (UID: \"49f9b7f2-8cc7-48eb-9906-c9bea914073a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.561724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.561572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvb7v\" (UniqueName: \"kubernetes.io/projected/49f9b7f2-8cc7-48eb-9906-c9bea914073a-kube-api-access-bvb7v\") pod \"cert-manager-webhook-6486ffd997-857qn\" (UID: \"49f9b7f2-8cc7-48eb-9906-c9bea914073a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.662596 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.662547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f9b7f2-8cc7-48eb-9906-c9bea914073a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-857qn\" (UID: \"49f9b7f2-8cc7-48eb-9906-c9bea914073a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.662750 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.662643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvb7v\" (UniqueName: \"kubernetes.io/projected/49f9b7f2-8cc7-48eb-9906-c9bea914073a-kube-api-access-bvb7v\") pod \"cert-manager-webhook-6486ffd997-857qn\" (UID: \"49f9b7f2-8cc7-48eb-9906-c9bea914073a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.675195 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.675165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f9b7f2-8cc7-48eb-9906-c9bea914073a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-857qn\" (UID: \"49f9b7f2-8cc7-48eb-9906-c9bea914073a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.675301 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.675247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvb7v\" (UniqueName: \"kubernetes.io/projected/49f9b7f2-8cc7-48eb-9906-c9bea914073a-kube-api-access-bvb7v\") pod \"cert-manager-webhook-6486ffd997-857qn\" (UID: \"49f9b7f2-8cc7-48eb-9906-c9bea914073a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:30.769334 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:30.769255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:31.055807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:31.055726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-857qn"]
Sep 10 13:35:31.061875 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:35:31.061850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f9b7f2_8cc7_48eb_9906_c9bea914073a.slice/crio-d9336ecfbe49255de7cfd7d4d3d3cfd7ab335746e585d0c396685ca63208b6f4 WatchSource:0}: Error finding container d9336ecfbe49255de7cfd7d4d3d3cfd7ab335746e585d0c396685ca63208b6f4: Status 404 returned error can't find the container with id d9336ecfbe49255de7cfd7d4d3d3cfd7ab335746e585d0c396685ca63208b6f4
Sep 10 13:35:31.723685 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:31.723650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn" event={"ID":"49f9b7f2-8cc7-48eb-9906-c9bea914073a","Type":"ContainerStarted","Data":"d9336ecfbe49255de7cfd7d4d3d3cfd7ab335746e585d0c396685ca63208b6f4"}
Sep 10 13:35:35.742438 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:35.742400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn" event={"ID":"49f9b7f2-8cc7-48eb-9906-c9bea914073a","Type":"ContainerStarted","Data":"6b571166a15df1f6b48fc395707c24926fd386f27a2f00b94e898f20d037635a"}
Sep 10 13:35:35.742811 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:35.742454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:35.761910 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:35.761868    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn" podStartSLOduration=1.275100485 podStartE2EDuration="5.761855306s" totalImagesPullingTime="4.486754821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:35.760848817 +0000 UTC m=+279.785663521" watchObservedRunningTime="2026-09-10 13:35:35.761855306 +0000 UTC m=+279.786670009"
Sep 10 13:35:36.846537 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:36.846502    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"]
Sep 10 13:35:36.849037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:36.849016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:36.851724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:36.851705    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fpnk7\""
Sep 10 13:35:36.860286 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:36.860263    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"]
Sep 10 13:35:37.020635 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.020570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp525\" (UniqueName: \"kubernetes.io/projected/4d89054e-0841-46e2-916b-f443b304b4e0-kube-api-access-rp525\") pod \"cert-manager-cainjector-559cdd8c87-dbq78\" (UID: \"4d89054e-0841-46e2-916b-f443b304b4e0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.020805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.020718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d89054e-0841-46e2-916b-f443b304b4e0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dbq78\" (UID: \"4d89054e-0841-46e2-916b-f443b304b4e0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.121654 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.121541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp525\" (UniqueName: \"kubernetes.io/projected/4d89054e-0841-46e2-916b-f443b304b4e0-kube-api-access-rp525\") pod \"cert-manager-cainjector-559cdd8c87-dbq78\" (UID: \"4d89054e-0841-46e2-916b-f443b304b4e0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.121654 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.121645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d89054e-0841-46e2-916b-f443b304b4e0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dbq78\" (UID: \"4d89054e-0841-46e2-916b-f443b304b4e0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.131033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.131000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d89054e-0841-46e2-916b-f443b304b4e0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dbq78\" (UID: \"4d89054e-0841-46e2-916b-f443b304b4e0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.131169 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.131090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp525\" (UniqueName: \"kubernetes.io/projected/4d89054e-0841-46e2-916b-f443b304b4e0-kube-api-access-rp525\") pod \"cert-manager-cainjector-559cdd8c87-dbq78\" (UID: \"4d89054e-0841-46e2-916b-f443b304b4e0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.158033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.158011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"
Sep 10 13:35:37.429080 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.428998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dbq78"]
Sep 10 13:35:37.432643 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:35:37.432618    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d89054e_0841_46e2_916b_f443b304b4e0.slice/crio-8648998073c14919fb00dcc1f6b4def6ccf113b5932b8814ab8bc42ed247bb68 WatchSource:0}: Error finding container 8648998073c14919fb00dcc1f6b4def6ccf113b5932b8814ab8bc42ed247bb68: Status 404 returned error can't find the container with id 8648998073c14919fb00dcc1f6b4def6ccf113b5932b8814ab8bc42ed247bb68
Sep 10 13:35:37.751717 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.751635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78" event={"ID":"4d89054e-0841-46e2-916b-f443b304b4e0","Type":"ContainerStarted","Data":"ff8fc30221a484603a73765170eefbd2488be484b15118db2ca077e8cc25335d"}
Sep 10 13:35:37.751717 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.751677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78" event={"ID":"4d89054e-0841-46e2-916b-f443b304b4e0","Type":"ContainerStarted","Data":"8648998073c14919fb00dcc1f6b4def6ccf113b5932b8814ab8bc42ed247bb68"}
Sep 10 13:35:37.773282 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:37.773230    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dbq78" podStartSLOduration=1.773214899 podStartE2EDuration="1.773214899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:37.772341819 +0000 UTC m=+281.797156522" watchObservedRunningTime="2026-09-10 13:35:37.773214899 +0000 UTC m=+281.798029602"
Sep 10 13:35:41.749326 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:41.749291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-857qn"
Sep 10 13:35:56.376934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:56.376900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:35:56.382841 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:56.382815    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:35:56.383700 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:35:56.383682    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 13:36:08.775228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.775192    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"]
Sep 10 13:36:08.779261 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.779241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.783283 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.783262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sf2hc\""
Sep 10 13:36:08.783798 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.783777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 13:36:08.783922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.783903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 13:36:08.784522 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.784507    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 13:36:08.784716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.784700    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 13:36:08.801827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.801798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"]
Sep 10 13:36:08.893737 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.893702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65d0d623-a0b6-4041-9202-a2037a64c1f1-tmp\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.893903 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.893742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tck\" (UniqueName: \"kubernetes.io/projected/65d0d623-a0b6-4041-9202-a2037a64c1f1-kube-api-access-x4tck\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.893903 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.893787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65d0d623-a0b6-4041-9202-a2037a64c1f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.893903 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.893826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65d0d623-a0b6-4041-9202-a2037a64c1f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.893903 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.893856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65d0d623-a0b6-4041-9202-a2037a64c1f1-manifests\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.994697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.994659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65d0d623-a0b6-4041-9202-a2037a64c1f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.994697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.994700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65d0d623-a0b6-4041-9202-a2037a64c1f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.994899 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.994720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65d0d623-a0b6-4041-9202-a2037a64c1f1-manifests\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.994899 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.994770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65d0d623-a0b6-4041-9202-a2037a64c1f1-tmp\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.994899 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.994794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tck\" (UniqueName: \"kubernetes.io/projected/65d0d623-a0b6-4041-9202-a2037a64c1f1-kube-api-access-x4tck\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.995229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.995203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65d0d623-a0b6-4041-9202-a2037a64c1f1-manifests\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.997163 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.997133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65d0d623-a0b6-4041-9202-a2037a64c1f1-tmp\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.997253 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.997234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65d0d623-a0b6-4041-9202-a2037a64c1f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:08.997311 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:08.997296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65d0d623-a0b6-4041-9202-a2037a64c1f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:09.003510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:09.003492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tck\" (UniqueName: \"kubernetes.io/projected/65d0d623-a0b6-4041-9202-a2037a64c1f1-kube-api-access-x4tck\") pod \"opendatahub-operator-controller-manager-94d598c87-hhc9q\" (UID: \"65d0d623-a0b6-4041-9202-a2037a64c1f1\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:09.090600 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:09.090546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:09.390099 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:09.390073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"]
Sep 10 13:36:09.393030 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:36:09.393002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d0d623_a0b6_4041_9202_a2037a64c1f1.slice/crio-bc8b5af5cd0be513f3150c30dcff992cbdf8fb4236fe22c6e313bf8a98022d93 WatchSource:0}: Error finding container bc8b5af5cd0be513f3150c30dcff992cbdf8fb4236fe22c6e313bf8a98022d93: Status 404 returned error can't find the container with id bc8b5af5cd0be513f3150c30dcff992cbdf8fb4236fe22c6e313bf8a98022d93
Sep 10 13:36:09.395083 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:09.395057    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:36:09.882705 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:09.882664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q" event={"ID":"65d0d623-a0b6-4041-9202-a2037a64c1f1","Type":"ContainerStarted","Data":"bc8b5af5cd0be513f3150c30dcff992cbdf8fb4236fe22c6e313bf8a98022d93"}
Sep 10 13:36:13.904542 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:13.904503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q" event={"ID":"65d0d623-a0b6-4041-9202-a2037a64c1f1","Type":"ContainerStarted","Data":"2c9ce0f0001869a0346f21982b47196a8cb884e932997a73642f110128cbae78"}
Sep 10 13:36:14.910512 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:14.910470    2577 generic.go:356] "Generic (PLEG): container finished" podID="65d0d623-a0b6-4041-9202-a2037a64c1f1" containerID="2c9ce0f0001869a0346f21982b47196a8cb884e932997a73642f110128cbae78" exitCode=0
Sep 10 13:36:14.910972 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:14.910555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q" event={"ID":"65d0d623-a0b6-4041-9202-a2037a64c1f1","Type":"ContainerDied","Data":"2c9ce0f0001869a0346f21982b47196a8cb884e932997a73642f110128cbae78"}
Sep 10 13:36:15.918083 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:15.918048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q" event={"ID":"65d0d623-a0b6-4041-9202-a2037a64c1f1","Type":"ContainerStarted","Data":"50c19ea10a1822c825d6916a91aae8c73a21893b47b4ab79fc6219c84fd5b074"}
Sep 10 13:36:15.918460 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:15.918204    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:15.989143 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:15.989091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q" podStartSLOduration=2.720657007 podStartE2EDuration="7.989074283s" totalImagesPullingTime="4.261814572s" totalInitContainerRuntime="1.006602704s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:15.986793794 +0000 UTC m=+320.011608498" watchObservedRunningTime="2026-09-10 13:36:15.989074283 +0000 UTC m=+320.013888985"
Sep 10 13:36:26.924410 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:26.924377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-hhc9q"
Sep 10 13:36:39.585683 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.585649    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"]
Sep 10 13:36:39.589308 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.589289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.591665 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.591632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 13:36:39.591665 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.591646    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 13:36:39.591665 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.591654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hc8pz\""
Sep 10 13:36:39.592146 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.592131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 13:36:39.601181 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.601148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"]
Sep 10 13:36:39.666738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666738 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ltt\" (UniqueName: \"kubernetes.io/projected/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-kube-api-access-s7ltt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666942 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666942 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666942 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666942 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666942 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.666942 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.666941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.667158 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.667003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.767631 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.767787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.767787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.767787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.767787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.767787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.767977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768035 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.768026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768181 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.768055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ltt\" (UniqueName: \"kubernetes.io/projected/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-kube-api-access-s7ltt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768251 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.768205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768349 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.768326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768467 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.768449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.768592 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.768558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.770031 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.770011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.770398 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.770377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.778643 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.778617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.797956 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.797929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ltt\" (UniqueName: \"kubernetes.io/projected/d20f11b0-7ada-4bd5-a6b5-e93d2b58994b-kube-api-access-s7ltt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k\" (UID: \"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:39.899145 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:39.899086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:40.177477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.177328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"]
Sep 10 13:36:40.179811 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:36:40.179773    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20f11b0_7ada_4bd5_a6b5_e93d2b58994b.slice/crio-570639cad11d50f7e3632301aea0d6d9a3567e2e33cc2d85c9d9d2fdba90cb8d WatchSource:0}: Error finding container 570639cad11d50f7e3632301aea0d6d9a3567e2e33cc2d85c9d9d2fdba90cb8d: Status 404 returned error can't find the container with id 570639cad11d50f7e3632301aea0d6d9a3567e2e33cc2d85c9d9d2fdba90cb8d
Sep 10 13:36:40.867380 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.867348    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mgnch"]
Sep 10 13:36:40.871785 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.871765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:40.875884 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.875862    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 13:36:40.876003 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.875907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbml\" (UniqueName: \"kubernetes.io/projected/9447ffbd-881b-4b7b-9380-7abcd2c3bdfb-kube-api-access-lzbml\") pod \"model-serving-api-54d89bd5fc-mgnch\" (UID: \"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:40.876060 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.876001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9447ffbd-881b-4b7b-9380-7abcd2c3bdfb-tls-certs\") pod \"model-serving-api-54d89bd5fc-mgnch\" (UID: \"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:40.884451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.884433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mgnch"]
Sep 10 13:36:40.976816 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.976790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9447ffbd-881b-4b7b-9380-7abcd2c3bdfb-tls-certs\") pod \"model-serving-api-54d89bd5fc-mgnch\" (UID: \"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:40.976909 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.976851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbml\" (UniqueName: \"kubernetes.io/projected/9447ffbd-881b-4b7b-9380-7abcd2c3bdfb-kube-api-access-lzbml\") pod \"model-serving-api-54d89bd5fc-mgnch\" (UID: \"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:40.979142 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.979123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9447ffbd-881b-4b7b-9380-7abcd2c3bdfb-tls-certs\") pod \"model-serving-api-54d89bd5fc-mgnch\" (UID: \"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:40.986151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:40.986128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbml\" (UniqueName: \"kubernetes.io/projected/9447ffbd-881b-4b7b-9380-7abcd2c3bdfb-kube-api-access-lzbml\") pod \"model-serving-api-54d89bd5fc-mgnch\" (UID: \"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:41.027303 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:41.027275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k" event={"ID":"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b","Type":"ContainerStarted","Data":"570639cad11d50f7e3632301aea0d6d9a3567e2e33cc2d85c9d9d2fdba90cb8d"}
Sep 10 13:36:41.182200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:41.181950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:41.465186 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:41.465116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mgnch"]
Sep 10 13:36:41.477547 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:36:41.477514    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9447ffbd_881b_4b7b_9380_7abcd2c3bdfb.slice/crio-e4253949f79dd05d1a443d34f27624e94e6228eaa997d1145625ac083394210f WatchSource:0}: Error finding container e4253949f79dd05d1a443d34f27624e94e6228eaa997d1145625ac083394210f: Status 404 returned error can't find the container with id e4253949f79dd05d1a443d34f27624e94e6228eaa997d1145625ac083394210f
Sep 10 13:36:42.033869 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:42.033832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch" event={"ID":"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb","Type":"ContainerStarted","Data":"e4253949f79dd05d1a443d34f27624e94e6228eaa997d1145625ac083394210f"}
Sep 10 13:36:43.057761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:43.057724    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:36:43.058008 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:43.057795    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:36:43.058008 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:43.057825    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:36:44.045880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:44.045844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k" event={"ID":"d20f11b0-7ada-4bd5-a6b5-e93d2b58994b","Type":"ContainerStarted","Data":"2ab6155d634684b93af2a7359a5c2e5748da649637d09c18c12539353231937e"}
Sep 10 13:36:44.082708 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:44.082657    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k" podStartSLOduration=2.207192183 podStartE2EDuration="5.082641819s" totalImagesPullingTime="2.875449636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:44.080878669 +0000 UTC m=+348.105693373" watchObservedRunningTime="2026-09-10 13:36:44.082641819 +0000 UTC m=+348.107456521"
Sep 10 13:36:44.899655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:44.899621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:44.904646 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:44.904624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:45.051080 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:45.051055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch" event={"ID":"9447ffbd-881b-4b7b-9380-7abcd2c3bdfb","Type":"ContainerStarted","Data":"c0d1d2869812bdc40d1357c1d8e230116e7d65df00c6fdd340cb4dae49029bd4"}
Sep 10 13:36:45.051247 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:45.051222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:45.052173 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:45.052151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k"
Sep 10 13:36:45.088140 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:45.088088    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch" podStartSLOduration=1.621968168 podStartE2EDuration="5.088072648s" totalImagesPullingTime="3.46610448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:45.087809138 +0000 UTC m=+349.112623842" watchObservedRunningTime="2026-09-10 13:36:45.088072648 +0000 UTC m=+349.112887350"
Sep 10 13:36:46.056745 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:46.056671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:36:48.275207 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.275167    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"]
Sep 10 13:36:48.283541 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.283520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.287884 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.287861    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 13:36:48.287988 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.287884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 13:36:48.288760 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.288736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 13:36:48.288854 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.288808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:36:48.288854 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.288818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nlz28\""
Sep 10 13:36:48.288954 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.288923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 13:36:48.294353 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.294332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"]
Sep 10 13:36:48.344847 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.344816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed44035f-cb33-44c2-9419-db0ed9fafadd-manager-config\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.344847 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.344851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfzl\" (UniqueName: \"kubernetes.io/projected/ed44035f-cb33-44c2-9419-db0ed9fafadd-kube-api-access-5cfzl\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.345056 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.344884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed44035f-cb33-44c2-9419-db0ed9fafadd-cert\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.345056 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.344973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed44035f-cb33-44c2-9419-db0ed9fafadd-metrics-cert\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.446249 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.446211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed44035f-cb33-44c2-9419-db0ed9fafadd-manager-config\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.446249 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.446249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfzl\" (UniqueName: \"kubernetes.io/projected/ed44035f-cb33-44c2-9419-db0ed9fafadd-kube-api-access-5cfzl\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.446502 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.446278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed44035f-cb33-44c2-9419-db0ed9fafadd-cert\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.446502 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.446317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed44035f-cb33-44c2-9419-db0ed9fafadd-metrics-cert\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.447305 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.447279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed44035f-cb33-44c2-9419-db0ed9fafadd-manager-config\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.449231 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.449208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed44035f-cb33-44c2-9419-db0ed9fafadd-metrics-cert\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.449328 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.449278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed44035f-cb33-44c2-9419-db0ed9fafadd-cert\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.455468 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.455446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfzl\" (UniqueName: \"kubernetes.io/projected/ed44035f-cb33-44c2-9419-db0ed9fafadd-kube-api-access-5cfzl\") pod \"lws-controller-manager-59689cfc58-79cns\" (UID: \"ed44035f-cb33-44c2-9419-db0ed9fafadd\") " pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.593279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.593245    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:48.864821 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:48.864798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"]
Sep 10 13:36:48.866848 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:36:48.866822    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded44035f_cb33_44c2_9419_db0ed9fafadd.slice/crio-a743c0dfc0011c21b2e5c5a6b28dd65972e090924899afb1f60d7d099090e21e WatchSource:0}: Error finding container a743c0dfc0011c21b2e5c5a6b28dd65972e090924899afb1f60d7d099090e21e: Status 404 returned error can't find the container with id a743c0dfc0011c21b2e5c5a6b28dd65972e090924899afb1f60d7d099090e21e
Sep 10 13:36:49.070503 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:49.070471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns" event={"ID":"ed44035f-cb33-44c2-9419-db0ed9fafadd","Type":"ContainerStarted","Data":"a743c0dfc0011c21b2e5c5a6b28dd65972e090924899afb1f60d7d099090e21e"}
Sep 10 13:36:50.788406 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.788374    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"]
Sep 10 13:36:50.796671 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.796645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"]
Sep 10 13:36:50.796818 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.796771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:50.799743 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.799721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.799866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.799796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 13:36:50.800127 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.800110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ktb5x\""
Sep 10 13:36:50.802014 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.801996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 13:36:50.804077 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.804057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jnl5h\""
Sep 10 13:36:50.807638 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.807619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"]
Sep 10 13:36:50.815310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.815277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"]
Sep 10 13:36:50.869058 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.869028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/833dcd84-968d-4ffb-8b5d-7f86109bcaf8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l5frs\" (UID: \"833dcd84-968d-4ffb-8b5d-7f86109bcaf8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.869216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.869154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92516cc8-609a-4371-9a79-3543d5f5bbde-cert\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:50.869216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.869191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdn9\" (UniqueName: \"kubernetes.io/projected/833dcd84-968d-4ffb-8b5d-7f86109bcaf8-kube-api-access-kqdn9\") pod \"llmisvc-controller-manager-777c6bd7d5-l5frs\" (UID: \"833dcd84-968d-4ffb-8b5d-7f86109bcaf8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.869346 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.869299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbjd\" (UniqueName: \"kubernetes.io/projected/92516cc8-609a-4371-9a79-3543d5f5bbde-kube-api-access-rwbjd\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:50.970496 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.970461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbjd\" (UniqueName: \"kubernetes.io/projected/92516cc8-609a-4371-9a79-3543d5f5bbde-kube-api-access-rwbjd\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:50.970695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.970504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/833dcd84-968d-4ffb-8b5d-7f86109bcaf8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l5frs\" (UID: \"833dcd84-968d-4ffb-8b5d-7f86109bcaf8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.970695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.970610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92516cc8-609a-4371-9a79-3543d5f5bbde-cert\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:50.970695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.970644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdn9\" (UniqueName: \"kubernetes.io/projected/833dcd84-968d-4ffb-8b5d-7f86109bcaf8-kube-api-access-kqdn9\") pod \"llmisvc-controller-manager-777c6bd7d5-l5frs\" (UID: \"833dcd84-968d-4ffb-8b5d-7f86109bcaf8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.970871 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:36:50.970765    2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 13:36:50.970871 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:36:50.970835    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92516cc8-609a-4371-9a79-3543d5f5bbde-cert podName:92516cc8-609a-4371-9a79-3543d5f5bbde nodeName:}" failed. No retries permitted until 2026-09-10 13:36:51.470814996 +0000 UTC m=+355.495629683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/92516cc8-609a-4371-9a79-3543d5f5bbde-cert") pod "kserve-controller-manager-d6f4b65b8-xbfl8" (UID: "92516cc8-609a-4371-9a79-3543d5f5bbde") : secret "kserve-webhook-server-cert" not found
Sep 10 13:36:50.973156 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.973128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/833dcd84-968d-4ffb-8b5d-7f86109bcaf8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l5frs\" (UID: \"833dcd84-968d-4ffb-8b5d-7f86109bcaf8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.985736 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.985707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdn9\" (UniqueName: \"kubernetes.io/projected/833dcd84-968d-4ffb-8b5d-7f86109bcaf8-kube-api-access-kqdn9\") pod \"llmisvc-controller-manager-777c6bd7d5-l5frs\" (UID: \"833dcd84-968d-4ffb-8b5d-7f86109bcaf8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:50.988230 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:50.988202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbjd\" (UniqueName: \"kubernetes.io/projected/92516cc8-609a-4371-9a79-3543d5f5bbde-kube-api-access-rwbjd\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:51.119588 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:51.119551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:51.467399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:51.467303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"]
Sep 10 13:36:51.470942 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:36:51.470918    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod833dcd84_968d_4ffb_8b5d_7f86109bcaf8.slice/crio-6c1ae92d1dbbcda305e4da24566fe74c1b67dfa66666923f83be87cdcea10159 WatchSource:0}: Error finding container 6c1ae92d1dbbcda305e4da24566fe74c1b67dfa66666923f83be87cdcea10159: Status 404 returned error can't find the container with id 6c1ae92d1dbbcda305e4da24566fe74c1b67dfa66666923f83be87cdcea10159
Sep 10 13:36:51.475163 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:51.475135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92516cc8-609a-4371-9a79-3543d5f5bbde-cert\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:51.477759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:51.477734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92516cc8-609a-4371-9a79-3543d5f5bbde-cert\") pod \"kserve-controller-manager-d6f4b65b8-xbfl8\" (UID: \"92516cc8-609a-4371-9a79-3543d5f5bbde\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:51.711830 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:51.711801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:51.998479 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:51.998419    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"]
Sep 10 13:36:52.001684 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:36:52.001647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92516cc8_609a_4371_9a79_3543d5f5bbde.slice/crio-8cf3a8e1ed1d4f017ddc33534627c9f261c48852fa1939d8021cd0a3cb2c1c1e WatchSource:0}: Error finding container 8cf3a8e1ed1d4f017ddc33534627c9f261c48852fa1939d8021cd0a3cb2c1c1e: Status 404 returned error can't find the container with id 8cf3a8e1ed1d4f017ddc33534627c9f261c48852fa1939d8021cd0a3cb2c1c1e
Sep 10 13:36:52.088911 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:52.088877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs" event={"ID":"833dcd84-968d-4ffb-8b5d-7f86109bcaf8","Type":"ContainerStarted","Data":"6c1ae92d1dbbcda305e4da24566fe74c1b67dfa66666923f83be87cdcea10159"}
Sep 10 13:36:52.090363 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:52.090333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns" event={"ID":"ed44035f-cb33-44c2-9419-db0ed9fafadd","Type":"ContainerStarted","Data":"8d4e73d2e974f78df9d50e1f405749ea298fa41ccede35df66c2744a4dd65890"}
Sep 10 13:36:52.090492 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:52.090387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:36:52.091451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:52.091430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8" event={"ID":"92516cc8-609a-4371-9a79-3543d5f5bbde","Type":"ContainerStarted","Data":"8cf3a8e1ed1d4f017ddc33534627c9f261c48852fa1939d8021cd0a3cb2c1c1e"}
Sep 10 13:36:52.126685 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:52.126631    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns" podStartSLOduration=1.78848619 podStartE2EDuration="4.126612075s" totalImagesPullingTime="2.338125885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:52.123003968 +0000 UTC m=+356.147818671" watchObservedRunningTime="2026-09-10 13:36:52.126612075 +0000 UTC m=+356.151426778"
Sep 10 13:36:55.111898 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:55.111853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs" event={"ID":"833dcd84-968d-4ffb-8b5d-7f86109bcaf8","Type":"ContainerStarted","Data":"2570f719ab998d1a74b0371d7a6d0570128d616cebe446ec7f5a0e71877e513f"}
Sep 10 13:36:55.112329 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:55.111932    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:36:55.113366 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:55.113343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8" event={"ID":"92516cc8-609a-4371-9a79-3543d5f5bbde","Type":"ContainerStarted","Data":"334ac85ea13dafbae70cb95dfddc97379db3e47db02570cc3bb634c9a5e8f2a9"}
Sep 10 13:36:55.113483 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:55.113463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:36:55.134309 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:55.134267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs" podStartSLOduration=1.990927028 podStartE2EDuration="5.134252329s" totalImagesPullingTime="3.143325301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:55.13300879 +0000 UTC m=+359.157823494" watchObservedRunningTime="2026-09-10 13:36:55.134252329 +0000 UTC m=+359.159067028"
Sep 10 13:36:55.154979 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:55.154935    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8" podStartSLOduration=2.540574503 podStartE2EDuration="5.154922565s" totalImagesPullingTime="2.614348062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:55.152737302 +0000 UTC m=+359.177552005" watchObservedRunningTime="2026-09-10 13:36:55.154922565 +0000 UTC m=+359.179737267"
Sep 10 13:36:57.064613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:36:57.064564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mgnch"
Sep 10 13:37:03.099691 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:37:03.099658    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59689cfc58-79cns"
Sep 10 13:37:26.120450 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:37:26.120412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5frs"
Sep 10 13:37:26.123713 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:37:26.123685    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xbfl8"
Sep 10 13:38:05.314431 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.314398    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"]
Sep 10 13:38:05.322359 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.322333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.325295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.325273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sztmq\""
Sep 10 13:38:05.325425 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.325345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 13:38:05.325556 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.325542    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 13:38:05.330677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.330655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"]
Sep 10 13:38:05.362172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.362147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33bb1d36-24d5-4a4f-a884-025c23424a29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.362267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.362176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkmm\" (UniqueName: \"kubernetes.io/projected/33bb1d36-24d5-4a4f-a884-025c23424a29-kube-api-access-kvkmm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.462951 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.462927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33bb1d36-24d5-4a4f-a884-025c23424a29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.463060 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.462957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkmm\" (UniqueName: \"kubernetes.io/projected/33bb1d36-24d5-4a4f-a884-025c23424a29-kube-api-access-kvkmm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.463279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.463262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33bb1d36-24d5-4a4f-a884-025c23424a29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.472271 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.472246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkmm\" (UniqueName: \"kubernetes.io/projected/33bb1d36-24d5-4a4f-a884-025c23424a29-kube-api-access-kvkmm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.634033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.634011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:05.913951 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:05.913929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"]
Sep 10 13:38:05.915423 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:38:05.915398    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bb1d36_24d5_4a4f_a884_025c23424a29.slice/crio-da9a05fd279522936a3ec3c1e9d828779c73842251493ad9e3ecdace5495b412 WatchSource:0}: Error finding container da9a05fd279522936a3ec3c1e9d828779c73842251493ad9e3ecdace5495b412: Status 404 returned error can't find the container with id da9a05fd279522936a3ec3c1e9d828779c73842251493ad9e3ecdace5495b412
Sep 10 13:38:06.466173 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:06.466105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j" event={"ID":"33bb1d36-24d5-4a4f-a884-025c23424a29","Type":"ContainerStarted","Data":"da9a05fd279522936a3ec3c1e9d828779c73842251493ad9e3ecdace5495b412"}
Sep 10 13:38:10.489875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:10.489785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j" event={"ID":"33bb1d36-24d5-4a4f-a884-025c23424a29","Type":"ContainerStarted","Data":"197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706"}
Sep 10 13:38:10.489875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:10.489854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:10.514485 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:10.514437    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j" podStartSLOduration=1.310409515 podStartE2EDuration="5.514421048s" totalImagesPullingTime="4.204011533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:10.512737252 +0000 UTC m=+434.537551954" watchObservedRunningTime="2026-09-10 13:38:10.514421048 +0000 UTC m=+434.539235749"
Sep 10 13:38:21.497319 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:21.497247    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:22.856745 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:22.856709    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"]
Sep 10 13:38:22.860004 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:22.859986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:22.886486 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:22.886456    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"]
Sep 10 13:38:23.012220 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.012185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc52c51-a055-4182-8fdb-0e2980792d4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bpg2f\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.012366 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.012266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kstvp\" (UniqueName: \"kubernetes.io/projected/cfc52c51-a055-4182-8fdb-0e2980792d4a-kube-api-access-kstvp\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bpg2f\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.018847 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.018811    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"]
Sep 10 13:38:23.019067 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:38:23.019048    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-kstvp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f" podUID="cfc52c51-a055-4182-8fdb-0e2980792d4a"
Sep 10 13:38:23.113337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.113258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kstvp\" (UniqueName: \"kubernetes.io/projected/cfc52c51-a055-4182-8fdb-0e2980792d4a-kube-api-access-kstvp\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bpg2f\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.113337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.113317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc52c51-a055-4182-8fdb-0e2980792d4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bpg2f\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.113683 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.113665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc52c51-a055-4182-8fdb-0e2980792d4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bpg2f\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.134681 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.134648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kstvp\" (UniqueName: \"kubernetes.io/projected/cfc52c51-a055-4182-8fdb-0e2980792d4a-kube-api-access-kstvp\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bpg2f\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.548444 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.548383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.555977 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.555956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:23.718680 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.718657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kstvp\" (UniqueName: \"kubernetes.io/projected/cfc52c51-a055-4182-8fdb-0e2980792d4a-kube-api-access-kstvp\") pod \"cfc52c51-a055-4182-8fdb-0e2980792d4a\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") "
Sep 10 13:38:23.718802 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.718708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc52c51-a055-4182-8fdb-0e2980792d4a-extensions-socket-volume\") pod \"cfc52c51-a055-4182-8fdb-0e2980792d4a\" (UID: \"cfc52c51-a055-4182-8fdb-0e2980792d4a\") "
Sep 10 13:38:23.718987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.718955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc52c51-a055-4182-8fdb-0e2980792d4a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cfc52c51-a055-4182-8fdb-0e2980792d4a" (UID: "cfc52c51-a055-4182-8fdb-0e2980792d4a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:23.720482 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.720455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc52c51-a055-4182-8fdb-0e2980792d4a-kube-api-access-kstvp" (OuterVolumeSpecName: "kube-api-access-kstvp") pod "cfc52c51-a055-4182-8fdb-0e2980792d4a" (UID: "cfc52c51-a055-4182-8fdb-0e2980792d4a"). InnerVolumeSpecName "kube-api-access-kstvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:23.819827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.819803    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kstvp\" (UniqueName: \"kubernetes.io/projected/cfc52c51-a055-4182-8fdb-0e2980792d4a-kube-api-access-kstvp\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:38:23.819827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.819826    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc52c51-a055-4182-8fdb-0e2980792d4a-extensions-socket-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:38:23.943076 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.943044    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"]
Sep 10 13:38:23.948380 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.948356    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"]
Sep 10 13:38:23.975465 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.975429    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"]
Sep 10 13:38:23.975754 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.975709    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" containerName="manager" containerID="cri-o://197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706" gracePeriod=2
Sep 10 13:38:23.977890 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.977864    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"]
Sep 10 13:38:23.978182 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:38:23.977915    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" is forbidden: User \"system:node:ip-10-0-130-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-202.ec2.internal' and this object" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:23.990863 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.990802    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"]
Sep 10 13:38:23.991266 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.991248    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" containerName="manager"
Sep 10 13:38:23.991266 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.991266    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" containerName="manager"
Sep 10 13:38:23.991404 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.991354    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" containerName="manager"
Sep 10 13:38:23.994294 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:23.994273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.011859 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.011833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"]
Sep 10 13:38:24.021857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.021826    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"]
Sep 10 13:38:24.025833 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.025813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:24.028661 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.028638    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7m4wl\""
Sep 10 13:38:24.042752 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:38:24.042721    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" is forbidden: User \"system:node:ip-10-0-130-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-202.ec2.internal' and this object" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:24.043193 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.043170    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"]
Sep 10 13:38:24.045376 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:38:24.045350    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" is forbidden: User \"system:node:ip-10-0-130-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-202.ec2.internal' and this object" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:24.122811 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.122781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6rzch\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.122935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.122853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzz6\" (UniqueName: \"kubernetes.io/projected/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-kube-api-access-7pzz6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6rzch\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.122975 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.122931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfcxq\" (UniqueName: \"kubernetes.io/projected/449c08d0-c09c-4c86-9e43-926d524709cc-kube-api-access-dfcxq\") pod \"limitador-operator-controller-manager-74495c69f8-mwq8p\" (UID: \"449c08d0-c09c-4c86-9e43-926d524709cc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:24.223377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.223355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:24.223725 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.223704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzz6\" (UniqueName: \"kubernetes.io/projected/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-kube-api-access-7pzz6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6rzch\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.223772 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.223739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfcxq\" (UniqueName: \"kubernetes.io/projected/449c08d0-c09c-4c86-9e43-926d524709cc-kube-api-access-dfcxq\") pod \"limitador-operator-controller-manager-74495c69f8-mwq8p\" (UID: \"449c08d0-c09c-4c86-9e43-926d524709cc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:24.223939 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.223919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6rzch\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.224227 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.224212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6rzch\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.226310 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:38:24.226283    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lbx2j\" is forbidden: User \"system:node:ip-10-0-130-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-202.ec2.internal' and this object" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:24.233333 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.233304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfcxq\" (UniqueName: \"kubernetes.io/projected/449c08d0-c09c-4c86-9e43-926d524709cc-kube-api-access-dfcxq\") pod \"limitador-operator-controller-manager-74495c69f8-mwq8p\" (UID: \"449c08d0-c09c-4c86-9e43-926d524709cc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:24.233414 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.233350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzz6\" (UniqueName: \"kubernetes.io/projected/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-kube-api-access-7pzz6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6rzch\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.324450 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.324429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33bb1d36-24d5-4a4f-a884-025c23424a29-extensions-socket-volume\") pod \"33bb1d36-24d5-4a4f-a884-025c23424a29\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") "
Sep 10 13:38:24.324534 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.324470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvkmm\" (UniqueName: \"kubernetes.io/projected/33bb1d36-24d5-4a4f-a884-025c23424a29-kube-api-access-kvkmm\") pod \"33bb1d36-24d5-4a4f-a884-025c23424a29\" (UID: \"33bb1d36-24d5-4a4f-a884-025c23424a29\") "
Sep 10 13:38:24.324950 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.324927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bb1d36-24d5-4a4f-a884-025c23424a29-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "33bb1d36-24d5-4a4f-a884-025c23424a29" (UID: "33bb1d36-24d5-4a4f-a884-025c23424a29"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:24.326366 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.326340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bb1d36-24d5-4a4f-a884-025c23424a29-kube-api-access-kvkmm" (OuterVolumeSpecName: "kube-api-access-kvkmm") pod "33bb1d36-24d5-4a4f-a884-025c23424a29" (UID: "33bb1d36-24d5-4a4f-a884-025c23424a29"). InnerVolumeSpecName "kube-api-access-kvkmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:24.357799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.357753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:24.362779 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.362766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:24.425396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.425365    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvkmm\" (UniqueName: \"kubernetes.io/projected/33bb1d36-24d5-4a4f-a884-025c23424a29-kube-api-access-kvkmm\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:38:24.425396 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.425398    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33bb1d36-24d5-4a4f-a884-025c23424a29-extensions-socket-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:38:24.452960 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.452931    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bb1d36-24d5-4a4f-a884-025c23424a29" path="/var/lib/kubelet/pods/33bb1d36-24d5-4a4f-a884-025c23424a29/volumes"
Sep 10 13:38:24.453333 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.453319    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc52c51-a055-4182-8fdb-0e2980792d4a" path="/var/lib/kubelet/pods/cfc52c51-a055-4182-8fdb-0e2980792d4a/volumes"
Sep 10 13:38:24.554001 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.553974    2577 generic.go:356] "Generic (PLEG): container finished" podID="33bb1d36-24d5-4a4f-a884-025c23424a29" containerID="197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706" exitCode=0
Sep 10 13:38:24.554094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.554016    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lbx2j"
Sep 10 13:38:24.554094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.554073    2577 scope.go:117] "RemoveContainer" containerID="197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706"
Sep 10 13:38:24.554217 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.554082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bpg2f"
Sep 10 13:38:24.563018 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.562991    2577 scope.go:117] "RemoveContainer" containerID="197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706"
Sep 10 13:38:24.563290 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:38:24.563272    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706\": container with ID starting with 197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706 not found: ID does not exist" containerID="197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706"
Sep 10 13:38:24.563357 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.563296    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706"} err="failed to get container status \"197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706\": rpc error: code = NotFound desc = could not find container \"197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706\": container with ID starting with 197772d98e128cf6dda10c09919782a829017b7c648b18a03cb86ee556170706 not found: ID does not exist"
Sep 10 13:38:24.643813 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.643793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"]
Sep 10 13:38:24.644789 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:38:24.644757    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdfd47b_321f_4fee_8a30_0c34d2ba0c1a.slice/crio-3ad12789e13a4aed063fcecf1b4fc639791e009e6a09d065688aa6c1c867a830 WatchSource:0}: Error finding container 3ad12789e13a4aed063fcecf1b4fc639791e009e6a09d065688aa6c1c867a830: Status 404 returned error can't find the container with id 3ad12789e13a4aed063fcecf1b4fc639791e009e6a09d065688aa6c1c867a830
Sep 10 13:38:24.645521 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:24.645499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"]
Sep 10 13:38:24.646256 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:38:24.646237    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449c08d0_c09c_4c86_9e43_926d524709cc.slice/crio-4010373f37b8915fb873d3ce9eebaa90df854d7ff8e53ace07871304b1340239 WatchSource:0}: Error finding container 4010373f37b8915fb873d3ce9eebaa90df854d7ff8e53ace07871304b1340239: Status 404 returned error can't find the container with id 4010373f37b8915fb873d3ce9eebaa90df854d7ff8e53ace07871304b1340239
Sep 10 13:38:25.562533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:25.562411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch" event={"ID":"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a","Type":"ContainerStarted","Data":"3f19676740335af9a276b138fba6845cdfee3b26ed4bd344ce1cee11a0e471a5"}
Sep 10 13:38:25.562533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:25.562477    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:25.562533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:25.562502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch" event={"ID":"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a","Type":"ContainerStarted","Data":"3ad12789e13a4aed063fcecf1b4fc639791e009e6a09d065688aa6c1c867a830"}
Sep 10 13:38:25.563990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:25.563963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p" event={"ID":"449c08d0-c09c-4c86-9e43-926d524709cc","Type":"ContainerStarted","Data":"4010373f37b8915fb873d3ce9eebaa90df854d7ff8e53ace07871304b1340239"}
Sep 10 13:38:26.569532 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:26.569495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p" event={"ID":"449c08d0-c09c-4c86-9e43-926d524709cc","Type":"ContainerStarted","Data":"99aa7d176cc81fd5a0c19ea6364f022d44b3dd030c795235bc089eeae9094dfb"}
Sep 10 13:38:26.569993 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:26.569660    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:26.594647 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:26.594598    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch" podStartSLOduration=3.594566618 podStartE2EDuration="3.594566618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:25.621509641 +0000 UTC m=+449.646324344" watchObservedRunningTime="2026-09-10 13:38:26.594566618 +0000 UTC m=+450.619381322"
Sep 10 13:38:26.595851 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:26.595820    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p" podStartSLOduration=2.16036887 podStartE2EDuration="3.595805571s" totalImagesPullingTime="1.435436701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:26.592813147 +0000 UTC m=+450.617627849" watchObservedRunningTime="2026-09-10 13:38:26.595805571 +0000 UTC m=+450.620620274"
Sep 10 13:38:36.572319 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:36.572289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:37.578081 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:37.578053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mwq8p"
Sep 10 13:38:39.476712 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.476677    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"]
Sep 10 13:38:39.477098 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.476897    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch" podUID="3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" containerName="manager" containerID="cri-o://3f19676740335af9a276b138fba6845cdfee3b26ed4bd344ce1cee11a0e471a5" gracePeriod=10
Sep 10 13:38:39.633297 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.633257    2577 generic.go:356] "Generic (PLEG): container finished" podID="3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" containerID="3f19676740335af9a276b138fba6845cdfee3b26ed4bd344ce1cee11a0e471a5" exitCode=0
Sep 10 13:38:39.633461 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.633334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch" event={"ID":"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a","Type":"ContainerDied","Data":"3f19676740335af9a276b138fba6845cdfee3b26ed4bd344ce1cee11a0e471a5"}
Sep 10 13:38:39.732431 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.732378    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:39.755006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.754983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pzz6\" (UniqueName: \"kubernetes.io/projected/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-kube-api-access-7pzz6\") pod \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") "
Sep 10 13:38:39.755165 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.755060    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-extensions-socket-volume\") pod \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\" (UID: \"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a\") "
Sep 10 13:38:39.755419 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.755396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" (UID: "3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:39.757024 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.757004    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-kube-api-access-7pzz6" (OuterVolumeSpecName: "kube-api-access-7pzz6") pod "3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" (UID: "3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a"). InnerVolumeSpecName "kube-api-access-7pzz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:39.841773 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.841741    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"]
Sep 10 13:38:39.842196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.842178    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" containerName="manager"
Sep 10 13:38:39.842280 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.842199    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" containerName="manager"
Sep 10 13:38:39.842332 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.842310    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" containerName="manager"
Sep 10 13:38:39.845864 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.845845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:39.855700 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.855677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rpj8f\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:39.855795 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.855737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4mt\" (UniqueName: \"kubernetes.io/projected/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-kube-api-access-gk4mt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rpj8f\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:39.855837 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.855814    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pzz6\" (UniqueName: \"kubernetes.io/projected/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-kube-api-access-7pzz6\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:38:39.855837 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.855832    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a-extensions-socket-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:38:39.908544 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.908514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"]
Sep 10 13:38:39.956689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.956661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4mt\" (UniqueName: \"kubernetes.io/projected/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-kube-api-access-gk4mt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rpj8f\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:39.956827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.956811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rpj8f\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:39.957178 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.957162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rpj8f\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:39.980101 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:39.980069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4mt\" (UniqueName: \"kubernetes.io/projected/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-kube-api-access-gk4mt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rpj8f\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:40.156043 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.156005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:40.441533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.441509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"]
Sep 10 13:38:40.443464 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:38:40.443440    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f49ac24_0adf_4262_bfbd_d9d3dff9f403.slice/crio-deb87e6798ee36ae00497dea44705772fbc79beb366c313c816050cc4dc981f6 WatchSource:0}: Error finding container deb87e6798ee36ae00497dea44705772fbc79beb366c313c816050cc4dc981f6: Status 404 returned error can't find the container with id deb87e6798ee36ae00497dea44705772fbc79beb366c313c816050cc4dc981f6
Sep 10 13:38:40.639666 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.639640    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"
Sep 10 13:38:40.640082 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.639676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch" event={"ID":"3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a","Type":"ContainerDied","Data":"3ad12789e13a4aed063fcecf1b4fc639791e009e6a09d065688aa6c1c867a830"}
Sep 10 13:38:40.640082 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.639742    2577 scope.go:117] "RemoveContainer" containerID="3f19676740335af9a276b138fba6845cdfee3b26ed4bd344ce1cee11a0e471a5"
Sep 10 13:38:40.641786 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.641752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" event={"ID":"3f49ac24-0adf-4262-bfbd-d9d3dff9f403","Type":"ContainerStarted","Data":"42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413"}
Sep 10 13:38:40.641918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.641815    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:38:40.641918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.641842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" event={"ID":"3f49ac24-0adf-4262-bfbd-d9d3dff9f403","Type":"ContainerStarted","Data":"deb87e6798ee36ae00497dea44705772fbc79beb366c313c816050cc4dc981f6"}
Sep 10 13:38:40.693343 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.693260    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" podStartSLOduration=1.693243445 podStartE2EDuration="1.693243445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:40.687052043 +0000 UTC m=+464.711866745" watchObservedRunningTime="2026-09-10 13:38:40.693243445 +0000 UTC m=+464.718058147"
Sep 10 13:38:40.707533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.707512    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"]
Sep 10 13:38:40.716690 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:40.716667    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6rzch"]
Sep 10 13:38:42.451536 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:42.451500    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a" path="/var/lib/kubelet/pods/3cdfd47b-321f-4fee-8a30-0c34d2ba0c1a/volumes"
Sep 10 13:38:51.649988 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:38:51.649957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 13:39:46.160929 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.160896    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9955587-k9ctw"]
Sep 10 13:39:46.163492 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.163476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.170546 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.170524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lz67j\""
Sep 10 13:39:46.170546 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.170535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 13:39:46.170744 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.170553    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 13:39:46.170839 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.170825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 13:39:46.184260 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.184228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9955587-k9ctw"]
Sep 10 13:39:46.207238 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.207210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-metrics-certs\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.207372 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.207247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-webhook-cert\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.207372 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.207342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl8b\" (UniqueName: \"kubernetes.io/projected/31d93039-d87a-4620-96ec-8ba8eac4abe5-kube-api-access-tzl8b\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.308740 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.308704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl8b\" (UniqueName: \"kubernetes.io/projected/31d93039-d87a-4620-96ec-8ba8eac4abe5-kube-api-access-tzl8b\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.308914 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.308758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-metrics-certs\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.308914 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.308794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-webhook-cert\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.311203 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.311173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-metrics-certs\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.311353 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.311335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-webhook-cert\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.318967 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.318938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl8b\" (UniqueName: \"kubernetes.io/projected/31d93039-d87a-4620-96ec-8ba8eac4abe5-kube-api-access-tzl8b\") pod \"maas-controller-5b9955587-k9ctw\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") " pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.473952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.473873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:46.753165 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.753026    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9955587-k9ctw"]
Sep 10 13:39:46.755364 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:39:46.755338    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d93039_d87a_4620_96ec_8ba8eac4abe5.slice/crio-6819b3436f6cfb7a116a9180b02681048ed4cc925deb7e9fea37dc135958aa26 WatchSource:0}: Error finding container 6819b3436f6cfb7a116a9180b02681048ed4cc925deb7e9fea37dc135958aa26: Status 404 returned error can't find the container with id 6819b3436f6cfb7a116a9180b02681048ed4cc925deb7e9fea37dc135958aa26
Sep 10 13:39:46.960547 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:46.960509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-k9ctw" event={"ID":"31d93039-d87a-4620-96ec-8ba8eac4abe5","Type":"ContainerStarted","Data":"6819b3436f6cfb7a116a9180b02681048ed4cc925deb7e9fea37dc135958aa26"}
Sep 10 13:39:49.978196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:49.978156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-k9ctw" event={"ID":"31d93039-d87a-4620-96ec-8ba8eac4abe5","Type":"ContainerStarted","Data":"2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd"}
Sep 10 13:39:49.978606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:49.978243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:39:50.007194 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:39:50.007119    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9955587-k9ctw" podStartSLOduration=1.551629366 podStartE2EDuration="4.007102059s" totalImagesPullingTime="2.455472693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:50.005328937 +0000 UTC m=+534.030143641" watchObservedRunningTime="2026-09-10 13:39:50.007102059 +0000 UTC m=+534.031916761"
Sep 10 13:40:00.985947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:00.985909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 13:40:22.750978 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.750935    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"]
Sep 10 13:40:22.754504 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.754484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.758113 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.758085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 13:40:22.758113 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.758109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 13:40:22.758306 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.758132    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-97x99\""
Sep 10 13:40:22.758306 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.758131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 13:40:22.758491 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.758475    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 13:40:22.766014 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.765992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"]
Sep 10 13:40:22.840828 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.840794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.841019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.840837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c69857-115b-4bfd-9b2c-9a8c8199d397-openshift-service-ca\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.841019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.840868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-maas-api-tls\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.841019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.840921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khzkf\" (UniqueName: \"kubernetes.io/projected/28c69857-115b-4bfd-9b2c-9a8c8199d397-kube-api-access-khzkf\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.941888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.941851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.941888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.941895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c69857-115b-4bfd-9b2c-9a8c8199d397-openshift-service-ca\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.942139 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.941939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-maas-api-tls\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.942139 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.941987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzkf\" (UniqueName: \"kubernetes.io/projected/28c69857-115b-4bfd-9b2c-9a8c8199d397-kube-api-access-khzkf\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.942139 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:40:22.942009    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 10 13:40:22.942139 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:40:22.942086    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs podName:28c69857-115b-4bfd-9b2c-9a8c8199d397 nodeName:}" failed. No retries permitted until 2026-09-10 13:40:23.442065002 +0000 UTC m=+567.466879707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs") pod "maas-api-5fcf74d56b-nxltb" (UID: "28c69857-115b-4bfd-9b2c-9a8c8199d397") : secret "maas-api-metrics-tls" not found
Sep 10 13:40:22.942724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.942700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c69857-115b-4bfd-9b2c-9a8c8199d397-openshift-service-ca\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.944550 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.944525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-maas-api-tls\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:22.952413 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:22.952389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khzkf\" (UniqueName: \"kubernetes.io/projected/28c69857-115b-4bfd-9b2c-9a8c8199d397-kube-api-access-khzkf\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:23.446114 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:23.446078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:23.446291 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:40:23.446236    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 10 13:40:23.446334 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:40:23.446303    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs podName:28c69857-115b-4bfd-9b2c-9a8c8199d397 nodeName:}" failed. No retries permitted until 2026-09-10 13:40:24.446287376 +0000 UTC m=+568.471102062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs") pod "maas-api-5fcf74d56b-nxltb" (UID: "28c69857-115b-4bfd-9b2c-9a8c8199d397") : secret "maas-api-metrics-tls" not found
Sep 10 13:40:24.453926 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:24.453884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:24.457074 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:24.457040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") pod \"maas-api-5fcf74d56b-nxltb\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:24.565775 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:24.565731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:24.852094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:24.852069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"]
Sep 10 13:40:24.854877 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:40:24.854846    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c69857_115b_4bfd_9b2c_9a8c8199d397.slice/crio-d47bb3f56decb84f8bb426de6774b2831bc02821f0d5965584ba3634eea93f65 WatchSource:0}: Error finding container d47bb3f56decb84f8bb426de6774b2831bc02821f0d5965584ba3634eea93f65: Status 404 returned error can't find the container with id d47bb3f56decb84f8bb426de6774b2831bc02821f0d5965584ba3634eea93f65
Sep 10 13:40:25.148310 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:25.148224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb" event={"ID":"28c69857-115b-4bfd-9b2c-9a8c8199d397","Type":"ContainerStarted","Data":"d47bb3f56decb84f8bb426de6774b2831bc02821f0d5965584ba3634eea93f65"}
Sep 10 13:40:27.160560 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:27.160520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb" event={"ID":"28c69857-115b-4bfd-9b2c-9a8c8199d397","Type":"ContainerStarted","Data":"edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba"}
Sep 10 13:40:27.160970 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:27.160602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:27.185393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:27.185331    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb" podStartSLOduration=3.372283148 podStartE2EDuration="5.185315336s" totalImagesPullingTime="1.813032188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:27.1837782 +0000 UTC m=+571.208592904" watchObservedRunningTime="2026-09-10 13:40:27.185315336 +0000 UTC m=+571.210130039"
Sep 10 13:40:33.172047 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:33.172019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 13:40:52.015730 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.015697    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"]
Sep 10 13:40:52.019141 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.019121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.023156 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.023137    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 13:40:52.023251 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.023139    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 13:40:52.023251 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.023146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 13:40:52.028386 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.028367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zm5zz\""
Sep 10 13:40:52.061046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.061022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"]
Sep 10 13:40:52.110666 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.110640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.110757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.110671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.110757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.110706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.110757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.110728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.110858 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.110839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.110892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.110869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vkw\" (UniqueName: \"kubernetes.io/projected/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-kube-api-access-q5vkw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211410 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vkw\" (UniqueName: \"kubernetes.io/projected/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-kube-api-access-q5vkw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211723 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211917 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.211990 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.211971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.213972 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.213952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.214245 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.214225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.225860 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.225839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vkw\" (UniqueName: \"kubernetes.io/projected/fa0b1a0c-e74b-4328-98cc-a15d1aa40c62-kube-api-access-q5vkw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9d67v\" (UID: \"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.329084 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.329059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:40:52.605881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:52.605857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"]
Sep 10 13:40:52.607859 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:40:52.607835    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0b1a0c_e74b_4328_98cc_a15d1aa40c62.slice/crio-ce3bca7a50c30cf48be66dddb0d9fe99fa9861356d9b8efb58751c5bfcabc1e6 WatchSource:0}: Error finding container ce3bca7a50c30cf48be66dddb0d9fe99fa9861356d9b8efb58751c5bfcabc1e6: Status 404 returned error can't find the container with id ce3bca7a50c30cf48be66dddb0d9fe99fa9861356d9b8efb58751c5bfcabc1e6
Sep 10 13:40:53.294835 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:53.294799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v" event={"ID":"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62","Type":"ContainerStarted","Data":"ce3bca7a50c30cf48be66dddb0d9fe99fa9861356d9b8efb58751c5bfcabc1e6"}
Sep 10 13:40:54.301682 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:54.301640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v" event={"ID":"fa0b1a0c-e74b-4328-98cc-a15d1aa40c62","Type":"ContainerStarted","Data":"d16a951b742871d08b33d9d696c77b95fb1c32696d36b4b0a01130a18ea2691a"}
Sep 10 13:40:54.326428 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:54.326375    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v" podStartSLOduration=2.316799343 podStartE2EDuration="3.326359495s" totalImagesPullingTime="1.009560152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:54.324367593 +0000 UTC m=+598.349182296" watchObservedRunningTime="2026-09-10 13:40:54.326359495 +0000 UTC m=+598.351174235"
Sep 10 13:40:56.415312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:56.415285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:40:56.417969 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:40:56.417947    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:41:02.329428 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:02.329379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:41:02.329962 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:02.329460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:41:02.341824 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:02.341798    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:41:02.353873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:02.353844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9d67v"
Sep 10 13:41:42.361572 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.361473    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"]
Sep 10 13:41:42.364295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.364275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.372469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.372441    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 13:41:42.378186 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.378158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"]
Sep 10 13:41:42.465701 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.465663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b50b7f8-fd3a-45cd-8306-22d62e48c864-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.465873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.465722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.465873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.465790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2cs\" (UniqueName: \"kubernetes.io/projected/9b50b7f8-fd3a-45cd-8306-22d62e48c864-kube-api-access-tf2cs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.465873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.465837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.465983 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.465954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.465983 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.465975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.567473 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.567430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b50b7f8-fd3a-45cd-8306-22d62e48c864-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.567724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.567498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.567724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.567516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2cs\" (UniqueName: \"kubernetes.io/projected/9b50b7f8-fd3a-45cd-8306-22d62e48c864-kube-api-access-tf2cs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.567724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.567612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.567933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.567824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.567933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.567853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.568049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.568024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.568436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.568406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.568436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.568421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.570139 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.570114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b50b7f8-fd3a-45cd-8306-22d62e48c864-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.570222 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.570124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b50b7f8-fd3a-45cd-8306-22d62e48c864-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.580849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.580821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2cs\" (UniqueName: \"kubernetes.io/projected/9b50b7f8-fd3a-45cd-8306-22d62e48c864-kube-api-access-tf2cs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m\" (UID: \"9b50b7f8-fd3a-45cd-8306-22d62e48c864\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.676215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.676107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:42.954649 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.954547    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"]
Sep 10 13:41:42.957690 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:41:42.957664    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b50b7f8_fd3a_45cd_8306_22d62e48c864.slice/crio-3d8ca42901e80c193d4d3947742fd4c25f126bce63346e677904949333ec4c8a WatchSource:0}: Error finding container 3d8ca42901e80c193d4d3947742fd4c25f126bce63346e677904949333ec4c8a: Status 404 returned error can't find the container with id 3d8ca42901e80c193d4d3947742fd4c25f126bce63346e677904949333ec4c8a
Sep 10 13:41:42.958989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:42.958970    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:41:43.557172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:43.557078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m" event={"ID":"9b50b7f8-fd3a-45cd-8306-22d62e48c864","Type":"ContainerStarted","Data":"b9d77171602326b5346194d0a54c42989938ef8eded7f9bb430b58cb1a0ff42a"}
Sep 10 13:41:43.557602 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:43.557176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m" event={"ID":"9b50b7f8-fd3a-45cd-8306-22d62e48c864","Type":"ContainerStarted","Data":"3d8ca42901e80c193d4d3947742fd4c25f126bce63346e677904949333ec4c8a"}
Sep 10 13:41:43.585606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:43.585530    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m" podStartSLOduration=1.360893973 podStartE2EDuration="1.585514706s" totalImagesPullingTime="224.620733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:43.584835483 +0000 UTC m=+647.609650187" watchObservedRunningTime="2026-09-10 13:41:43.585514706 +0000 UTC m=+647.610329408"
Sep 10 13:41:52.676734 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:52.676697    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:52.677184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:52.676814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:52.688699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:52.688671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:53.617412 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:53.617386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m"
Sep 10 13:41:56.393347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.393310    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"]
Sep 10 13:41:56.396662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.396642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.411963 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.411940    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 13:41:56.423529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.423488    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"]
Sep 10 13:41:56.490922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.490891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.491071 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.490935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.491071 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.490984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.491071 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.491031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.491216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.491134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xkh\" (UniqueName: \"kubernetes.io/projected/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-kube-api-access-l6xkh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.491216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.491194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.591748 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.591717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.591934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.591788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xkh\" (UniqueName: \"kubernetes.io/projected/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-kube-api-access-l6xkh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.591934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.591832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.591934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.591921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.592115 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.592023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.592115 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.592069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.592401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.592323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.592401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.592397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.592648 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.592494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.594608 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.594514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.594750 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.594729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.601220 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.601200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xkh\" (UniqueName: \"kubernetes.io/projected/9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0-kube-api-access-l6xkh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjx72\" (UID: \"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.706933 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.706848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:41:56.984124 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:56.984040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"]
Sep 10 13:41:56.987164 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:41:56.987126    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df2ceb3_9df0_4dda_a406_a2ecaf1d2bf0.slice/crio-d102102cc431239100bb3a194837cf8d76533fb6fd072511c7f8c019fe5fa558 WatchSource:0}: Error finding container d102102cc431239100bb3a194837cf8d76533fb6fd072511c7f8c019fe5fa558: Status 404 returned error can't find the container with id d102102cc431239100bb3a194837cf8d76533fb6fd072511c7f8c019fe5fa558
Sep 10 13:41:57.627483 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:57.627446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72" event={"ID":"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0","Type":"ContainerStarted","Data":"3970e6a138d5aedf47eca9654227abfd9822588ff93b2341620877d247195818"}
Sep 10 13:41:57.627847 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:57.627501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72" event={"ID":"9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0","Type":"ContainerStarted","Data":"d102102cc431239100bb3a194837cf8d76533fb6fd072511c7f8c019fe5fa558"}
Sep 10 13:41:57.655170 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:41:57.655122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72" podStartSLOduration=1.377088516 podStartE2EDuration="1.655088892s" totalImagesPullingTime="278.000376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:57.653228802 +0000 UTC m=+661.678043515" watchObservedRunningTime="2026-09-10 13:41:57.655088892 +0000 UTC m=+661.679903594"
Sep 10 13:42:06.707256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:42:06.707210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:42:06.707761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:42:06.707340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:42:06.720214 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:42:06.720192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:42:07.687724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:42:07.687688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjx72"
Sep 10 13:43:40.301180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.301106    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"]
Sep 10 13:43:40.305431 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.305407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.307790 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.307766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f1d168-serving-cert\""
Sep 10 13:43:40.320517 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.320487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"]
Sep 10 13:43:40.380052 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.380015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-metrics-certs\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.380208 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.380065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.380280 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.380234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-maas-api-tls\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.380340 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.380312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz8xv\" (UniqueName: \"kubernetes.io/projected/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-kube-api-access-jz8xv\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.481815 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.481784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-metrics-certs\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.481815 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.481820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.482064 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.481900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-maas-api-tls\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.482064 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.481919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz8xv\" (UniqueName: \"kubernetes.io/projected/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-kube-api-access-jz8xv\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.482603 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.482565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.484254 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.484228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-maas-api-tls\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.484380 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.484277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-metrics-certs\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.491722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.491698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz8xv\" (UniqueName: \"kubernetes.io/projected/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-kube-api-access-jz8xv\") pod \"maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.616534 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.616508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:40.891952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:40.891917    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"]
Sep 10 13:43:40.894391 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:43:40.894350    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7d5a9f_69c9_47c9_ad05_92ff062fc400.slice/crio-f19af41bb98a1083d5b741640d1edb3849fad17fb79a52277d3054fe8d7e9520 WatchSource:0}: Error finding container f19af41bb98a1083d5b741640d1edb3849fad17fb79a52277d3054fe8d7e9520: Status 404 returned error can't find the container with id f19af41bb98a1083d5b741640d1edb3849fad17fb79a52277d3054fe8d7e9520
Sep 10 13:43:41.147892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:41.147814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5" event={"ID":"bb7d5a9f-69c9-47c9-ad05-92ff062fc400","Type":"ContainerStarted","Data":"f19af41bb98a1083d5b741640d1edb3849fad17fb79a52277d3054fe8d7e9520"}
Sep 10 13:43:42.154901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:42.154867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5" event={"ID":"bb7d5a9f-69c9-47c9-ad05-92ff062fc400","Type":"ContainerStarted","Data":"90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439"}
Sep 10 13:43:42.155274 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:42.154998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:42.176954 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:42.176898    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5" podStartSLOduration=1.64696808 podStartE2EDuration="2.176880996s" totalImagesPullingTime="529.912916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:42.174333556 +0000 UTC m=+766.199148272" watchObservedRunningTime="2026-09-10 13:43:42.176880996 +0000 UTC m=+766.201695699"
Sep 10 13:43:46.851848 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.851813    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"]
Sep 10 13:43:46.855528 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.855510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.858241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.858217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ba18dc-openshift-default-dockercfg-4mhgm\""
Sep 10 13:43:46.875495 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.875468    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"]
Sep 10 13:43:46.947861 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.947829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-envoy\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.947861 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.947858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istiod-ca-cert\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.947884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-podinfo\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.947914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-token\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.947955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-credential-socket\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.948008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjlg\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-kube-api-access-4cjlg\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.948092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-socket\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.948125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-certs\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:46.948172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:46.948146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-data\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.048620 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjlg\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-kube-api-access-4cjlg\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.048780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-socket\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.048780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-certs\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.048780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-data\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.048780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-envoy\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.048780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istiod-ca-cert\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049054 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-podinfo\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049054 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-token\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049054 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.048871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-credential-socket\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049226 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.049165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-certs\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049226 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.049213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-socket\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049342 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.049275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-data\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.049427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-credential-socket\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.049615 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.049569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istiod-ca-cert\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.051196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.051174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-envoy\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.051409 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.051389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-podinfo\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.064476 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.064438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-token\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.065766 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.065746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjlg\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-kube-api-access-4cjlg\") pod \"e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") " pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.167175 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.167097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:47.454922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.454885    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"]
Sep 10 13:43:47.456814 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:43:47.456792    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3eaa3a_c4d4_4d85_8f3b_d8cb96d8e1e7.slice/crio-1205c3da201de55ff67471c995733c5ff4fd6fbccc45d4a0dde9564dbd963b59 WatchSource:0}: Error finding container 1205c3da201de55ff67471c995733c5ff4fd6fbccc45d4a0dde9564dbd963b59: Status 404 returned error can't find the container with id 1205c3da201de55ff67471c995733c5ff4fd6fbccc45d4a0dde9564dbd963b59
Sep 10 13:43:47.458877 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.458842    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:43:47.458978 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.458920    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:43:47.458978 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:47.458962    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:43:48.166377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:48.166348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:43:48.190539 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:48.190505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" event={"ID":"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7","Type":"ContainerStarted","Data":"2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"}
Sep 10 13:43:48.190685 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:48.190561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" event={"ID":"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7","Type":"ContainerStarted","Data":"1205c3da201de55ff67471c995733c5ff4fd6fbccc45d4a0dde9564dbd963b59"}
Sep 10 13:43:48.310865 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:48.310803    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podStartSLOduration=2.310779889 podStartE2EDuration="2.310779889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:48.291976464 +0000 UTC m=+772.316791167" watchObservedRunningTime="2026-09-10 13:43:48.310779889 +0000 UTC m=+772.335594593"
Sep 10 13:43:49.168208 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:49.168174    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:49.173780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:49.173752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:49.196007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:49.195980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:49.196853 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:49.196834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:43:50.394184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.394147    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"]
Sep 10 13:43:50.398499 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.398474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.400863 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.400844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e9a01385-serving-cert\""
Sep 10 13:43:50.412437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.412415    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"]
Sep 10 13:43:50.481837 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.481804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-metrics-certs\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.481995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.481900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fd3b60-0975-4778-869a-154e4a14745d-openshift-service-ca\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.481995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.481955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwr7v\" (UniqueName: \"kubernetes.io/projected/f3fd3b60-0975-4778-869a-154e4a14745d-kube-api-access-gwr7v\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.481995 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.481991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.583130 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.583096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwr7v\" (UniqueName: \"kubernetes.io/projected/f3fd3b60-0975-4778-869a-154e4a14745d-kube-api-access-gwr7v\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.583130 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.583134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.583341 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.583205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-metrics-certs\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.583341 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.583253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fd3b60-0975-4778-869a-154e4a14745d-openshift-service-ca\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.583341 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:43:50.583306    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-serving-cert: secret "maas-api-e2e-ait-e9a01385-serving-cert" not found
Sep 10 13:43:50.583461 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:43:50.583392    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls podName:f3fd3b60-0975-4778-869a-154e4a14745d nodeName:}" failed. No retries permitted until 2026-09-10 13:43:51.083376227 +0000 UTC m=+775.108190907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls") pod "maas-api-e2e-ait-e9a01385-c5849874c-blcgv" (UID: "f3fd3b60-0975-4778-869a-154e4a14745d") : secret "maas-api-e2e-ait-e9a01385-serving-cert" not found
Sep 10 13:43:50.584001 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.583984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fd3b60-0975-4778-869a-154e4a14745d-openshift-service-ca\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.585664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.585641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-metrics-certs\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:50.592202 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:50.592179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwr7v\" (UniqueName: \"kubernetes.io/projected/f3fd3b60-0975-4778-869a-154e4a14745d-kube-api-access-gwr7v\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:51.088032 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:51.088002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:51.090509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:51.090482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls\") pod \"maas-api-e2e-ait-e9a01385-c5849874c-blcgv\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:51.315026 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:51.314998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:51.607973 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:51.607945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"]
Sep 10 13:43:51.609313 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:43:51.609287    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fd3b60_0975_4778_869a_154e4a14745d.slice/crio-432644b5a493b7e9eb9b7b59a44e8950bb7bc5415a6714de719dd9e7f5fb08aa WatchSource:0}: Error finding container 432644b5a493b7e9eb9b7b59a44e8950bb7bc5415a6714de719dd9e7f5fb08aa: Status 404 returned error can't find the container with id 432644b5a493b7e9eb9b7b59a44e8950bb7bc5415a6714de719dd9e7f5fb08aa
Sep 10 13:43:52.218270 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.218229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv" event={"ID":"f3fd3b60-0975-4778-869a-154e4a14745d","Type":"ContainerStarted","Data":"2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359"}
Sep 10 13:43:52.218416 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.218312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv" event={"ID":"f3fd3b60-0975-4778-869a-154e4a14745d","Type":"ContainerStarted","Data":"432644b5a493b7e9eb9b7b59a44e8950bb7bc5415a6714de719dd9e7f5fb08aa"}
Sep 10 13:43:52.218416 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.218383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:43:52.240932 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.240884    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv" podStartSLOduration=1.825791409 podStartE2EDuration="2.240870228s" totalImagesPullingTime="415.078819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:52.238312546 +0000 UTC m=+776.263127250" watchObservedRunningTime="2026-09-10 13:43:52.240870228 +0000 UTC m=+776.265684931"
Sep 10 13:43:52.738007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.737972    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"]
Sep 10 13:43:52.741789 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.741767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.745094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.745075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ba18dc-serving-cert\""
Sep 10 13:43:52.803395 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.803367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.803548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.803483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhw42\" (UniqueName: \"kubernetes.io/projected/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-kube-api-access-hhw42\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.803548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.803541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-metrics-certs\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.803648 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.803563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-openshift-service-ca\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.810275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.810249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"]
Sep 10 13:43:52.904910 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.904868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.905070 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.904999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhw42\" (UniqueName: \"kubernetes.io/projected/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-kube-api-access-hhw42\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.905070 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:43:52.905038    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-serving-cert: secret "maas-api-e2e-autores-ba18dc-serving-cert" not found
Sep 10 13:43:52.905070 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.905065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-metrics-certs\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.905229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.905091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-openshift-service-ca\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.905229 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:43:52.905111    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls podName:8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e nodeName:}" failed. No retries permitted until 2026-09-10 13:43:53.405089501 +0000 UTC m=+777.429904185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls") pod "maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" (UID: "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e") : secret "maas-api-e2e-autores-ba18dc-serving-cert" not found
Sep 10 13:43:52.905818 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.905799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-openshift-service-ca\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.907647 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.907626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-metrics-certs\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:52.914451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:52.914422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhw42\" (UniqueName: \"kubernetes.io/projected/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-kube-api-access-hhw42\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:53.410482 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:53.410450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:53.412864 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:53.412841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls\") pod \"maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:53.651417 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:53.651390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:53.933869 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:53.933845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"]
Sep 10 13:43:53.935694 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:43:53.935662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c87d9ba_d0ff_4364_8609_6b5d3dcacf0e.slice/crio-97fd6f7466f5b505ca154711cf16a771f051c74201285d04b83ebeea7274580d WatchSource:0}: Error finding container 97fd6f7466f5b505ca154711cf16a771f051c74201285d04b83ebeea7274580d: Status 404 returned error can't find the container with id 97fd6f7466f5b505ca154711cf16a771f051c74201285d04b83ebeea7274580d
Sep 10 13:43:54.231951 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:54.231861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" event={"ID":"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e","Type":"ContainerStarted","Data":"97fd6f7466f5b505ca154711cf16a771f051c74201285d04b83ebeea7274580d"}
Sep 10 13:43:55.238358 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:55.238323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" event={"ID":"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e","Type":"ContainerStarted","Data":"f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd"}
Sep 10 13:43:55.238801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:55.238444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:43:55.269154 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:55.269107    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" podStartSLOduration=2.918132013 podStartE2EDuration="3.269092591s" totalImagesPullingTime="350.960578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:55.267688795 +0000 UTC m=+779.292503509" watchObservedRunningTime="2026-09-10 13:43:55.269092591 +0000 UTC m=+779.293907315"
Sep 10 13:43:58.241037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:43:58.240978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:44:01.249899 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:01.249871    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:44:03.844000 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.843964    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"]
Sep 10 13:44:03.848072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.848052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:03.851744 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.851722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e6d037-serving-cert\""
Sep 10 13:44:03.864922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.864901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"]
Sep 10 13:44:03.918669 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.918633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk92f\" (UniqueName: \"kubernetes.io/projected/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-kube-api-access-gk92f\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:03.918911 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.918811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:03.918911 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.918842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-metrics-certs\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:03.918911 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:03.918883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.019709 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.019667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk92f\" (UniqueName: \"kubernetes.io/projected/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-kube-api-access-gk92f\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.019965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.019814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.019965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.019841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-metrics-certs\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.019965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.019883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.020157 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:44:04.020073    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-serving-cert: secret "maas-api-e2e-inf-b-e6d037-serving-cert" not found
Sep 10 13:44:04.020157 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:44:04.020136    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls podName:abce2590-c3ea-4603-bb3c-0e8538b8f5c2 nodeName:}" failed. No retries permitted until 2026-09-10 13:44:04.520116621 +0000 UTC m=+788.544931322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls") pod "maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" (UID: "abce2590-c3ea-4603-bb3c-0e8538b8f5c2") : secret "maas-api-e2e-inf-b-e6d037-serving-cert" not found
Sep 10 13:44:04.020844 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.020814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.023371 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.023321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-metrics-certs\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.031953 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.031928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk92f\" (UniqueName: \"kubernetes.io/projected/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-kube-api-access-gk92f\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.526332 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.526298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.529153 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.529122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:04.759880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:04.759844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:05.052389 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:05.052361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"]
Sep 10 13:44:05.054412 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:44:05.054374    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabce2590_c3ea_4603_bb3c_0e8538b8f5c2.slice/crio-cf324ed80330252740c8bca23f400044bf77828a87bc777eeba4cd2ee5a826be WatchSource:0}: Error finding container cf324ed80330252740c8bca23f400044bf77828a87bc777eeba4cd2ee5a826be: Status 404 returned error can't find the container with id cf324ed80330252740c8bca23f400044bf77828a87bc777eeba4cd2ee5a826be
Sep 10 13:44:05.237754 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:05.237711    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 13:44:05.237908 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:05.237780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 13:44:05.297610 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:05.297556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" event={"ID":"abce2590-c3ea-4603-bb3c-0e8538b8f5c2","Type":"ContainerStarted","Data":"cf324ed80330252740c8bca23f400044bf77828a87bc777eeba4cd2ee5a826be"}
Sep 10 13:44:06.307732 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:06.307686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" event={"ID":"abce2590-c3ea-4603-bb3c-0e8538b8f5c2","Type":"ContainerStarted","Data":"8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450"}
Sep 10 13:44:06.308186 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:06.307996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:07.314436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.314403    2577 generic.go:356] "Generic (PLEG): container finished" podID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerID="2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff" exitCode=0
Sep 10 13:44:07.314930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.314478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" event={"ID":"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7","Type":"ContainerDied","Data":"2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"}
Sep 10 13:44:07.315100 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.315076    2577 scope.go:117] "RemoveContainer" containerID="2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"
Sep 10 13:44:07.316223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.316193    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:44:07.316291 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.316278    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:44:07.316333 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.316318    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:44:07.348855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:07.348802    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" podStartSLOduration=3.958487722 podStartE2EDuration="4.348788303s" totalImagesPullingTime="390.300581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:06.346448839 +0000 UTC m=+790.371263544" watchObservedRunningTime="2026-09-10 13:44:07.348788303 +0000 UTC m=+791.373603022"
Sep 10 13:44:08.324289 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:08.324251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" event={"ID":"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7","Type":"ContainerStarted","Data":"d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494"}
Sep 10 13:44:09.168195 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:09.168156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:44:09.168437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:09.168414    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 13:44:09.168527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:09.168487    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 13:44:10.168156 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:10.168117    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 13:44:10.168655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:10.168174    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 13:44:11.167825 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:11.167793    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 13:44:11.168145 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:11.167853    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 13:44:12.167607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:12.167539    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 13:44:12.168068 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:12.167635    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 13:44:12.347556 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:12.347523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:44:13.167605 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:13.167554    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 10 13:44:13.167811 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:13.167624    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 10 13:44:14.171127 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:14.171089    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:44:14.171518 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:14.171371    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:44:14.172184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:14.172164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:44:14.967221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:14.967188    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"]
Sep 10 13:44:14.967436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:14.967392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv" podUID="f3fd3b60-0975-4778-869a-154e4a14745d" containerName="maas-api" containerID="cri-o://2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359" gracePeriod=30
Sep 10 13:44:15.213702 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.213677    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:44:15.341937 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.341906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls\") pod \"f3fd3b60-0975-4778-869a-154e4a14745d\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") "
Sep 10 13:44:15.342097 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.341951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fd3b60-0975-4778-869a-154e4a14745d-openshift-service-ca\") pod \"f3fd3b60-0975-4778-869a-154e4a14745d\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") "
Sep 10 13:44:15.342139 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.342126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwr7v\" (UniqueName: \"kubernetes.io/projected/f3fd3b60-0975-4778-869a-154e4a14745d-kube-api-access-gwr7v\") pod \"f3fd3b60-0975-4778-869a-154e4a14745d\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") "
Sep 10 13:44:15.342173 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.342163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-metrics-certs\") pod \"f3fd3b60-0975-4778-869a-154e4a14745d\" (UID: \"f3fd3b60-0975-4778-869a-154e4a14745d\") "
Sep 10 13:44:15.342328 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.342294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fd3b60-0975-4778-869a-154e4a14745d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3fd3b60-0975-4778-869a-154e4a14745d" (UID: "f3fd3b60-0975-4778-869a-154e4a14745d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:44:15.342590 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.342555    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fd3b60-0975-4778-869a-154e4a14745d-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:15.344447 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.344416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3fd3b60-0975-4778-869a-154e4a14745d" (UID: "f3fd3b60-0975-4778-869a-154e4a14745d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:44:15.344548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.344490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3fd3b60-0975-4778-869a-154e4a14745d" (UID: "f3fd3b60-0975-4778-869a-154e4a14745d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:44:15.344548 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.344514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fd3b60-0975-4778-869a-154e4a14745d-kube-api-access-gwr7v" (OuterVolumeSpecName: "kube-api-access-gwr7v") pod "f3fd3b60-0975-4778-869a-154e4a14745d" (UID: "f3fd3b60-0975-4778-869a-154e4a14745d"). InnerVolumeSpecName "kube-api-access-gwr7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:44:15.372771 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.372737    2577 generic.go:356] "Generic (PLEG): container finished" podID="f3fd3b60-0975-4778-869a-154e4a14745d" containerID="2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359" exitCode=0
Sep 10 13:44:15.372909 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.372791    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"
Sep 10 13:44:15.372909 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.372827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv" event={"ID":"f3fd3b60-0975-4778-869a-154e4a14745d","Type":"ContainerDied","Data":"2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359"}
Sep 10 13:44:15.372909 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.372891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv" event={"ID":"f3fd3b60-0975-4778-869a-154e4a14745d","Type":"ContainerDied","Data":"432644b5a493b7e9eb9b7b59a44e8950bb7bc5415a6714de719dd9e7f5fb08aa"}
Sep 10 13:44:15.373072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.372921    2577 scope.go:117] "RemoveContainer" containerID="2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359"
Sep 10 13:44:15.390186 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.389917    2577 scope.go:117] "RemoveContainer" containerID="2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359"
Sep 10 13:44:15.390520 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:44:15.390458    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359\": container with ID starting with 2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359 not found: ID does not exist" containerID="2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359"
Sep 10 13:44:15.390520 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.390506    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359"} err="failed to get container status \"2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359\": rpc error: code = NotFound desc = could not find container \"2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359\": container with ID starting with 2476d079dda89543678e39dc175df929e752a18b0a20191f39152665a70da359 not found: ID does not exist"
Sep 10 13:44:15.413924 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.413893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"]
Sep 10 13:44:15.436173 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.436147    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9a01385-c5849874c-blcgv"]
Sep 10 13:44:15.443955 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.443930    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:15.443955 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.443956    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwr7v\" (UniqueName: \"kubernetes.io/projected/f3fd3b60-0975-4778-869a-154e4a14745d-kube-api-access-gwr7v\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:15.444164 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:15.443969    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fd3b60-0975-4778-869a-154e4a14745d-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:16.452180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:16.452145    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3fd3b60-0975-4778-869a-154e4a14745d" path="/var/lib/kubelet/pods/f3fd3b60-0975-4778-869a-154e4a14745d/volumes"
Sep 10 13:44:25.429574 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.429485    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"]
Sep 10 13:44:25.430172 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.430153    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3fd3b60-0975-4778-869a-154e4a14745d" containerName="maas-api"
Sep 10 13:44:25.430267 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.430174    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3fd3b60-0975-4778-869a-154e4a14745d" containerName="maas-api"
Sep 10 13:44:25.430321 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.430309    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3fd3b60-0975-4778-869a-154e4a14745d" containerName="maas-api"
Sep 10 13:44:25.433930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.433911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.439658 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.439636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b1d82512-serving-cert\""
Sep 10 13:44:25.520851 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.520328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"]
Sep 10 13:44:25.545552 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.545514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.545724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.545574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120cd3f-19c0-4021-8015-849022f13384-openshift-service-ca\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.545724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.545653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-metrics-certs\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.545724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.545711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnrt\" (UniqueName: \"kubernetes.io/projected/6120cd3f-19c0-4021-8015-849022f13384-kube-api-access-vnnrt\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.646712 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.646678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnrt\" (UniqueName: \"kubernetes.io/projected/6120cd3f-19c0-4021-8015-849022f13384-kube-api-access-vnnrt\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.646880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.646732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.646880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.646780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120cd3f-19c0-4021-8015-849022f13384-openshift-service-ca\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.646964 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:44:25.646883    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-serving-cert: secret "maas-api-e2e-migrate-b1d82512-serving-cert" not found
Sep 10 13:44:25.646964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.646906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-metrics-certs\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.646964 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:44:25.646954    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls podName:6120cd3f-19c0-4021-8015-849022f13384 nodeName:}" failed. No retries permitted until 2026-09-10 13:44:26.146939046 +0000 UTC m=+810.171753726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls") pod "maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" (UID: "6120cd3f-19c0-4021-8015-849022f13384") : secret "maas-api-e2e-migrate-b1d82512-serving-cert" not found
Sep 10 13:44:25.647431 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.647413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120cd3f-19c0-4021-8015-849022f13384-openshift-service-ca\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.649307 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.649290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-metrics-certs\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:25.694101 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:25.694035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnrt\" (UniqueName: \"kubernetes.io/projected/6120cd3f-19c0-4021-8015-849022f13384-kube-api-access-vnnrt\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:26.150934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:26.150898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:26.153458 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:26.153431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls\") pod \"maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:26.346480 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:26.346445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:26.638277 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:26.638247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"]
Sep 10 13:44:26.638712 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:44:26.638568    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6120cd3f_19c0_4021_8015_849022f13384.slice/crio-20b25d5d5aa2f678f4232db28de372f06e201b6fa1b7cbd67de4e1a3c205bc72 WatchSource:0}: Error finding container 20b25d5d5aa2f678f4232db28de372f06e201b6fa1b7cbd67de4e1a3c205bc72: Status 404 returned error can't find the container with id 20b25d5d5aa2f678f4232db28de372f06e201b6fa1b7cbd67de4e1a3c205bc72
Sep 10 13:44:27.444046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:27.444003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" event={"ID":"6120cd3f-19c0-4021-8015-849022f13384","Type":"ContainerStarted","Data":"af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8"}
Sep 10 13:44:27.444248 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:27.444093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" event={"ID":"6120cd3f-19c0-4021-8015-849022f13384","Type":"ContainerStarted","Data":"20b25d5d5aa2f678f4232db28de372f06e201b6fa1b7cbd67de4e1a3c205bc72"}
Sep 10 13:44:27.444248 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:27.444168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:27.481078 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:27.481025    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" podStartSLOduration=2.099403742 podStartE2EDuration="2.481010044s" totalImagesPullingTime="381.606302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:27.477854511 +0000 UTC m=+811.502669213" watchObservedRunningTime="2026-09-10 13:44:27.481010044 +0000 UTC m=+811.505824746"
Sep 10 13:44:33.454597 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:33.454552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:50.926949 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:50.926910    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"]
Sep 10 13:44:50.927456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:50.927189    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" podUID="6120cd3f-19c0-4021-8015-849022f13384" containerName="maas-api" containerID="cri-o://af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8" gracePeriod=30
Sep 10 13:44:51.185877 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.185808    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:51.327801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.327378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-metrics-certs\") pod \"6120cd3f-19c0-4021-8015-849022f13384\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") "
Sep 10 13:44:51.327801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.327484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls\") pod \"6120cd3f-19c0-4021-8015-849022f13384\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") "
Sep 10 13:44:51.327801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.327544    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120cd3f-19c0-4021-8015-849022f13384-openshift-service-ca\") pod \"6120cd3f-19c0-4021-8015-849022f13384\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") "
Sep 10 13:44:51.327801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.327656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnnrt\" (UniqueName: \"kubernetes.io/projected/6120cd3f-19c0-4021-8015-849022f13384-kube-api-access-vnnrt\") pod \"6120cd3f-19c0-4021-8015-849022f13384\" (UID: \"6120cd3f-19c0-4021-8015-849022f13384\") "
Sep 10 13:44:51.328377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.328342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6120cd3f-19c0-4021-8015-849022f13384-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6120cd3f-19c0-4021-8015-849022f13384" (UID: "6120cd3f-19c0-4021-8015-849022f13384"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:44:51.330362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.330330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6120cd3f-19c0-4021-8015-849022f13384" (UID: "6120cd3f-19c0-4021-8015-849022f13384"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:44:51.330452 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.330385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6120cd3f-19c0-4021-8015-849022f13384" (UID: "6120cd3f-19c0-4021-8015-849022f13384"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:44:51.330497 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.330455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6120cd3f-19c0-4021-8015-849022f13384-kube-api-access-vnnrt" (OuterVolumeSpecName: "kube-api-access-vnnrt") pod "6120cd3f-19c0-4021-8015-849022f13384" (UID: "6120cd3f-19c0-4021-8015-849022f13384"). InnerVolumeSpecName "kube-api-access-vnnrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:44:51.429445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.429403    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120cd3f-19c0-4021-8015-849022f13384-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:51.429445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.429430    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnrt\" (UniqueName: \"kubernetes.io/projected/6120cd3f-19c0-4021-8015-849022f13384-kube-api-access-vnnrt\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:51.429445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.429440    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:51.429445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.429449    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6120cd3f-19c0-4021-8015-849022f13384-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:44:51.578144 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.578104    2577 generic.go:356] "Generic (PLEG): container finished" podID="6120cd3f-19c0-4021-8015-849022f13384" containerID="af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8" exitCode=0
Sep 10 13:44:51.578337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.578146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" event={"ID":"6120cd3f-19c0-4021-8015-849022f13384","Type":"ContainerDied","Data":"af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8"}
Sep 10 13:44:51.578337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.578225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2" event={"ID":"6120cd3f-19c0-4021-8015-849022f13384","Type":"ContainerDied","Data":"20b25d5d5aa2f678f4232db28de372f06e201b6fa1b7cbd67de4e1a3c205bc72"}
Sep 10 13:44:51.578337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.578257    2577 scope.go:117] "RemoveContainer" containerID="af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8"
Sep 10 13:44:51.578640 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.578620    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"
Sep 10 13:44:51.590187 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.590170    2577 scope.go:117] "RemoveContainer" containerID="af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8"
Sep 10 13:44:51.590662 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:44:51.590487    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8\": container with ID starting with af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8 not found: ID does not exist" containerID="af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8"
Sep 10 13:44:51.590662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.590522    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8"} err="failed to get container status \"af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8\": rpc error: code = NotFound desc = could not find container \"af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8\": container with ID starting with af81d54f8c1b28eb81b40964a648df3bc9918b4e719c4a1668af05c7dc8d56b8 not found: ID does not exist"
Sep 10 13:44:51.610174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.610150    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"]
Sep 10 13:44:51.616681 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:51.616658    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1d82512-8486ff99d9-rz2w2"]
Sep 10 13:44:52.453808 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:52.453772    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6120cd3f-19c0-4021-8015-849022f13384" path="/var/lib/kubelet/pods/6120cd3f-19c0-4021-8015-849022f13384/volumes"
Sep 10 13:44:56.696671 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.696634    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"]
Sep 10 13:44:56.697125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.697111    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6120cd3f-19c0-4021-8015-849022f13384" containerName="maas-api"
Sep 10 13:44:56.697125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.697126    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6120cd3f-19c0-4021-8015-849022f13384" containerName="maas-api"
Sep 10 13:44:56.697226 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.697216    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6120cd3f-19c0-4021-8015-849022f13384" containerName="maas-api"
Sep 10 13:44:56.701654 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.701637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.705111 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.705085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0d6a65c4-serving-cert\""
Sep 10 13:44:56.716875 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.716848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"]
Sep 10 13:44:56.781469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.781392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-maas-api-tls\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.781469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.781461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2m6\" (UniqueName: \"kubernetes.io/projected/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-kube-api-access-dv2m6\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.781678 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.781501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-openshift-service-ca\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.781678 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.781530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-metrics-certs\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.882564 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.882529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2m6\" (UniqueName: \"kubernetes.io/projected/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-kube-api-access-dv2m6\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.882778 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.882574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-openshift-service-ca\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.882778 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.882637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-metrics-certs\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.882778 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.882706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-maas-api-tls\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.883421 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.883394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-openshift-service-ca\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.885166 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.885144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-maas-api-tls\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.885253 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.885227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-metrics-certs\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:56.892999 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:56.892977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2m6\" (UniqueName: \"kubernetes.io/projected/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-kube-api-access-dv2m6\") pod \"maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:57.012671 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:57.012634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:57.296980 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:57.296955    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"]
Sep 10 13:44:57.299888 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:44:57.299841    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d70c8c8_84b6_4620_8651_9c58dddd1ecc.slice/crio-d8b350da203517b4179b663843be63a553ef24d9ab218c9983804b33f006ee17 WatchSource:0}: Error finding container d8b350da203517b4179b663843be63a553ef24d9ab218c9983804b33f006ee17: Status 404 returned error can't find the container with id d8b350da203517b4179b663843be63a553ef24d9ab218c9983804b33f006ee17
Sep 10 13:44:57.611218 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:57.611179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d" event={"ID":"5d70c8c8-84b6-4620-8651-9c58dddd1ecc","Type":"ContainerStarted","Data":"d8b350da203517b4179b663843be63a553ef24d9ab218c9983804b33f006ee17"}
Sep 10 13:44:58.618235 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:58.618202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d" event={"ID":"5d70c8c8-84b6-4620-8651-9c58dddd1ecc","Type":"ContainerStarted","Data":"dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795"}
Sep 10 13:44:58.618787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:58.618329    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:44:58.640525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:44:58.640463    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d" podStartSLOduration=2.176426493 podStartE2EDuration="2.640446079s" totalImagesPullingTime="464.019586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:58.639119283 +0000 UTC m=+842.663933986" watchObservedRunningTime="2026-09-10 13:44:58.640446079 +0000 UTC m=+842.665260840"
Sep 10 13:45:04.631127 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:04.631095    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:45:26.873322 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:26.873269    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"]
Sep 10 13:45:26.874364 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:26.873600    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d" podUID="5d70c8c8-84b6-4620-8651-9c58dddd1ecc" containerName="maas-api" containerID="cri-o://dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795" gracePeriod=30
Sep 10 13:45:27.150616 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.150573    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:45:27.199312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.199273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-maas-api-tls\") pod \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") "
Sep 10 13:45:27.199510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.199325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv2m6\" (UniqueName: \"kubernetes.io/projected/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-kube-api-access-dv2m6\") pod \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") "
Sep 10 13:45:27.199510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.199465    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-metrics-certs\") pod \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") "
Sep 10 13:45:27.199672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.199513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-openshift-service-ca\") pod \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\" (UID: \"5d70c8c8-84b6-4620-8651-9c58dddd1ecc\") "
Sep 10 13:45:27.199936 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.199903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d70c8c8-84b6-4620-8651-9c58dddd1ecc" (UID: "5d70c8c8-84b6-4620-8651-9c58dddd1ecc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:45:27.201949 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.201913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d70c8c8-84b6-4620-8651-9c58dddd1ecc" (UID: "5d70c8c8-84b6-4620-8651-9c58dddd1ecc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:45:27.202092 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.202068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5d70c8c8-84b6-4620-8651-9c58dddd1ecc" (UID: "5d70c8c8-84b6-4620-8651-9c58dddd1ecc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:45:27.202141 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.202076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-kube-api-access-dv2m6" (OuterVolumeSpecName: "kube-api-access-dv2m6") pod "5d70c8c8-84b6-4620-8651-9c58dddd1ecc" (UID: "5d70c8c8-84b6-4620-8651-9c58dddd1ecc"). InnerVolumeSpecName "kube-api-access-dv2m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:45:27.300415 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.300381    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:45:27.300415 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.300411    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2m6\" (UniqueName: \"kubernetes.io/projected/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-kube-api-access-dv2m6\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:45:27.300415 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.300421    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:45:27.300755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.300435    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70c8c8-84b6-4620-8651-9c58dddd1ecc-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:45:27.787288 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.787243    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d70c8c8-84b6-4620-8651-9c58dddd1ecc" containerID="dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795" exitCode=0
Sep 10 13:45:27.787288 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.787294    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"
Sep 10 13:45:27.787505 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.787324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d" event={"ID":"5d70c8c8-84b6-4620-8651-9c58dddd1ecc","Type":"ContainerDied","Data":"dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795"}
Sep 10 13:45:27.787505 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.787381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d" event={"ID":"5d70c8c8-84b6-4620-8651-9c58dddd1ecc","Type":"ContainerDied","Data":"d8b350da203517b4179b663843be63a553ef24d9ab218c9983804b33f006ee17"}
Sep 10 13:45:27.787505 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.787409    2577 scope.go:117] "RemoveContainer" containerID="dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795"
Sep 10 13:45:27.797557 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.797535    2577 scope.go:117] "RemoveContainer" containerID="dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795"
Sep 10 13:45:27.797903 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:45:27.797882    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795\": container with ID starting with dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795 not found: ID does not exist" containerID="dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795"
Sep 10 13:45:27.797985 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.797911    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795"} err="failed to get container status \"dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795\": rpc error: code = NotFound desc = could not find container \"dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795\": container with ID starting with dbb64c4c3c5b0f5f80371374bb9c413f97d7e89e1d015ddb34f1f6ce03e4e795 not found: ID does not exist"
Sep 10 13:45:27.816221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.816188    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"]
Sep 10 13:45:27.820542 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:27.820499    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d6a65c4-7d9bb5656b-fn54d"]
Sep 10 13:45:28.453527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:28.453493    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d70c8c8-84b6-4620-8651-9c58dddd1ecc" path="/var/lib/kubelet/pods/5d70c8c8-84b6-4620-8651-9c58dddd1ecc/volumes"
Sep 10 13:45:35.511320 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.511274    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"]
Sep 10 13:45:35.512045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.512022    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d70c8c8-84b6-4620-8651-9c58dddd1ecc" containerName="maas-api"
Sep 10 13:45:35.512139 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.512048    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d70c8c8-84b6-4620-8651-9c58dddd1ecc" containerName="maas-api"
Sep 10 13:45:35.512207 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.512174    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d70c8c8-84b6-4620-8651-9c58dddd1ecc" containerName="maas-api"
Sep 10 13:45:35.517158 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.517134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.524596 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.524445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e404ac71-serving-cert\""
Sep 10 13:45:35.526568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.526541    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"]
Sep 10 13:45:35.589485 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.589445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w667b\" (UniqueName: \"kubernetes.io/projected/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-kube-api-access-w667b\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.589699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.589553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.589699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.589608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-openshift-service-ca\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.589699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.589670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-metrics-certs\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.690657 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.690617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.690657 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.690657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-openshift-service-ca\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.690891 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.690713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-metrics-certs\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.690891 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:45:35.690780    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-serving-cert: secret "maas-api-e2e-derive-e404ac71-serving-cert" not found
Sep 10 13:45:35.690891 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.690797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w667b\" (UniqueName: \"kubernetes.io/projected/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-kube-api-access-w667b\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.690891 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:45:35.690850    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls podName:e9485fcc-88b9-4c13-b25f-2eb6c7eddd11 nodeName:}" failed. No retries permitted until 2026-09-10 13:45:36.190832697 +0000 UTC m=+880.215647377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls") pod "maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" (UID: "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11") : secret "maas-api-e2e-derive-e404ac71-serving-cert" not found
Sep 10 13:45:35.691387 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.691367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-openshift-service-ca\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.693229 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.693214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-metrics-certs\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:35.703606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:35.703564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w667b\" (UniqueName: \"kubernetes.io/projected/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-kube-api-access-w667b\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:36.195964 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:36.195916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:36.198370 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:36.198341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls\") pod \"maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:36.433447 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:36.433406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:36.727874 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:36.727009    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"]
Sep 10 13:45:36.740200 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:45:36.740121    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9485fcc_88b9_4c13_b25f_2eb6c7eddd11.slice/crio-d0769f9b33e905e1ecb6394ec2bfc6608f7f023bcbdc9c73834583ace7344f03 WatchSource:0}: Error finding container d0769f9b33e905e1ecb6394ec2bfc6608f7f023bcbdc9c73834583ace7344f03: Status 404 returned error can't find the container with id d0769f9b33e905e1ecb6394ec2bfc6608f7f023bcbdc9c73834583ace7344f03
Sep 10 13:45:36.845107 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:36.845066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" event={"ID":"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11","Type":"ContainerStarted","Data":"d0769f9b33e905e1ecb6394ec2bfc6608f7f023bcbdc9c73834583ace7344f03"}
Sep 10 13:45:37.852058 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:37.852021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" event={"ID":"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11","Type":"ContainerStarted","Data":"3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa"}
Sep 10 13:45:37.852466 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:37.852224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:37.873881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:37.873830    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" podStartSLOduration=2.5442567 podStartE2EDuration="2.873813189s" totalImagesPullingTime="329.556489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:37.872714489 +0000 UTC m=+881.897529193" watchObservedRunningTime="2026-09-10 13:45:37.873813189 +0000 UTC m=+881.898627910"
Sep 10 13:45:43.862420 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:43.862385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:45:56.460068 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:56.459986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:45:56.460506 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:45:56.460299    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:46:04.581959 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.581925    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"]
Sep 10 13:46:04.582436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.582125    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" podUID="8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" containerName="maas-api" containerID="cri-o://f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd" gracePeriod=30
Sep 10 13:46:04.836096 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.836038    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:46:04.905595 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.905561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-openshift-service-ca\") pod \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") "
Sep 10 13:46:04.905759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.905642    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-metrics-certs\") pod \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") "
Sep 10 13:46:04.905759 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.905753    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls\") pod \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") "
Sep 10 13:46:04.905886 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.905791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhw42\" (UniqueName: \"kubernetes.io/projected/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-kube-api-access-hhw42\") pod \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\" (UID: \"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e\") "
Sep 10 13:46:04.905886 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.905865    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" (UID: "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:04.906134 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.906118    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:04.907839 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.907806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-kube-api-access-hhw42" (OuterVolumeSpecName: "kube-api-access-hhw42") pod "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" (UID: "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e"). InnerVolumeSpecName "kube-api-access-hhw42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:04.907839 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.907832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" (UID: "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:04.907980 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.907806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" (UID: "8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:04.998216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.998183    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" containerID="f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd" exitCode=0
Sep 10 13:46:04.998404 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.998249    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"
Sep 10 13:46:04.998404 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.998282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" event={"ID":"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e","Type":"ContainerDied","Data":"f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd"}
Sep 10 13:46:04.998404 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.998338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9" event={"ID":"8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e","Type":"ContainerDied","Data":"97fd6f7466f5b505ca154711cf16a771f051c74201285d04b83ebeea7274580d"}
Sep 10 13:46:04.998404 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:04.998369    2577 scope.go:117] "RemoveContainer" containerID="f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd"
Sep 10 13:46:05.007831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.007809    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.007923 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.007839    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.007923 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.007854    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhw42\" (UniqueName: \"kubernetes.io/projected/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e-kube-api-access-hhw42\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.010740 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.010718    2577 scope.go:117] "RemoveContainer" containerID="f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd"
Sep 10 13:46:05.011015 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:46:05.010996    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd\": container with ID starting with f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd not found: ID does not exist" containerID="f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd"
Sep 10 13:46:05.011077 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.011022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd"} err="failed to get container status \"f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd\": rpc error: code = NotFound desc = could not find container \"f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd\": container with ID starting with f93562430845a77c52bfb94e2cba55a3a354c2e832ce544c1aea9b2e9e1143bd not found: ID does not exist"
Sep 10 13:46:05.066388 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.066354    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"]
Sep 10 13:46:05.068443 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.068417    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ba18dc-7f78bc789c-rwzp9"]
Sep 10 13:46:05.085925 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.085900    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"]
Sep 10 13:46:05.086155 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.086090    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" podUID="e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" containerName="maas-api" containerID="cri-o://3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa" gracePeriod=30
Sep 10 13:46:05.335959 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.335935    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:46:05.410976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.410898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-openshift-service-ca\") pod \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") "
Sep 10 13:46:05.410976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.410964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-metrics-certs\") pod \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") "
Sep 10 13:46:05.411223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.411048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w667b\" (UniqueName: \"kubernetes.io/projected/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-kube-api-access-w667b\") pod \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") "
Sep 10 13:46:05.411223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.411070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls\") pod \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\" (UID: \"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11\") "
Sep 10 13:46:05.411384 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.411352    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" (UID: "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:05.413210 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.413181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" (UID: "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:05.413479 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.413458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" (UID: "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:05.413571 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.413547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-kube-api-access-w667b" (OuterVolumeSpecName: "kube-api-access-w667b") pod "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" (UID: "e9485fcc-88b9-4c13-b25f-2eb6c7eddd11"). InnerVolumeSpecName "kube-api-access-w667b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:05.512076 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.512037    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.512076 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.512071    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.512076 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.512084    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w667b\" (UniqueName: \"kubernetes.io/projected/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-kube-api-access-w667b\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.512076 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:05.512096    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:06.005902 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.005860    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" containerID="3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa" exitCode=0
Sep 10 13:46:06.006362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.005927    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"
Sep 10 13:46:06.006362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.005932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" event={"ID":"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11","Type":"ContainerDied","Data":"3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa"}
Sep 10 13:46:06.006362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.006094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j" event={"ID":"e9485fcc-88b9-4c13-b25f-2eb6c7eddd11","Type":"ContainerDied","Data":"d0769f9b33e905e1ecb6394ec2bfc6608f7f023bcbdc9c73834583ace7344f03"}
Sep 10 13:46:06.006362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.006131    2577 scope.go:117] "RemoveContainer" containerID="3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa"
Sep 10 13:46:06.018513 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.018434    2577 scope.go:117] "RemoveContainer" containerID="3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa"
Sep 10 13:46:06.021770 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:46:06.020543    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa\": container with ID starting with 3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa not found: ID does not exist" containerID="3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa"
Sep 10 13:46:06.021770 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.020600    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa"} err="failed to get container status \"3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa\": rpc error: code = NotFound desc = could not find container \"3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa\": container with ID starting with 3a3f2946b87f1788a30fa343e7f8de294f5e13ed1ad97933a8058e6152c8d0fa not found: ID does not exist"
Sep 10 13:46:06.056874 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.056841    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"]
Sep 10 13:46:06.059448 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.059410    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e404ac71-77c89d5b89-v2c2j"]
Sep 10 13:46:06.453298 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.453260    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" path="/var/lib/kubelet/pods/8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e/volumes"
Sep 10 13:46:06.453805 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:06.453785    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" path="/var/lib/kubelet/pods/e9485fcc-88b9-4c13-b25f-2eb6c7eddd11/volumes"
Sep 10 13:46:30.300629 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.300598    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"]
Sep 10 13:46:30.301066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.301052    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" containerName="maas-api"
Sep 10 13:46:30.301066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.301067    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" containerName="maas-api"
Sep 10 13:46:30.301066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.301081    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" containerName="maas-api"
Sep 10 13:46:30.301066 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.301086    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" containerName="maas-api"
Sep 10 13:46:30.301235 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.301144    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c87d9ba-d0ff-4364-8609-6b5d3dcacf0e" containerName="maas-api"
Sep 10 13:46:30.301235 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.301154    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9485fcc-88b9-4c13-b25f-2eb6c7eddd11" containerName="maas-api"
Sep 10 13:46:30.304504 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.304483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.306943 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.306924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-3696db05-kserve-self-signed-certs\""
Sep 10 13:46:30.316390 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.316370    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"]
Sep 10 13:46:30.456276 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.456244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-home\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.456458 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.456363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-model-cache\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.456458 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.456394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tls-certs\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.456458 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.456438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-dshm\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.456640 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.456476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tmp-dir\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.456640 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.456503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvzm\" (UniqueName: \"kubernetes.io/projected/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-kube-api-access-lxvzm\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.557622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-home\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.557763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-model-cache\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.557763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tls-certs\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.557763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-dshm\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.557763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tmp-dir\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.557763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvzm\" (UniqueName: \"kubernetes.io/projected/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-kube-api-access-lxvzm\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.558038 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.557954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-home\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.558038 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.558031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-model-cache\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.558125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.558107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tmp-dir\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.559957 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.559933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-dshm\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.560184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.560169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tls-certs\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.567639 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.567616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvzm\" (UniqueName: \"kubernetes.io/projected/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-kube-api-access-lxvzm\") pod \"e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") " pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.613664 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.613631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:30.903953 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:30.903912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"]
Sep 10 13:46:30.906425 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:46:30.906395    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode21a39c1_d4f6_489e_bcf2_65d5b8756b6f.slice/crio-4b52f5526dba1e4eb87feea08602e6fec77030995b113c11eced457339a5e83f WatchSource:0}: Error finding container 4b52f5526dba1e4eb87feea08602e6fec77030995b113c11eced457339a5e83f: Status 404 returned error can't find the container with id 4b52f5526dba1e4eb87feea08602e6fec77030995b113c11eced457339a5e83f
Sep 10 13:46:31.141053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:31.141020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7" event={"ID":"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f","Type":"ContainerStarted","Data":"262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2"}
Sep 10 13:46:31.141053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:31.141060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7" event={"ID":"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f","Type":"ContainerStarted","Data":"4b52f5526dba1e4eb87feea08602e6fec77030995b113c11eced457339a5e83f"}
Sep 10 13:46:31.161286 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:31.161198    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7" podStartSLOduration=1.161184636 podStartE2EDuration="1.161184636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:31.159374893 +0000 UTC m=+935.184189594" watchObservedRunningTime="2026-09-10 13:46:31.161184636 +0000 UTC m=+935.185999339"
Sep 10 13:46:40.614521 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:40.614483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:40.614991 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:40.614534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:40.631901 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:40.631878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:41.204527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:41.204498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:46:41.499162 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:41.499080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"]
Sep 10 13:46:41.499316 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:41.499277    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" containerID="cri-o://d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494" gracePeriod=30
Sep 10 13:46:44.174195 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:44.174153    2577 patch_prober.go:28] interesting pod/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:46:44.174557 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:44.174217    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:46:46.753730 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.753703    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:46:46.812849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.812786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-podinfo\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.812849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.812843    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cjlg\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-kube-api-access-4cjlg\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.812879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-token\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.812908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-data\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.812946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-certs\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.812978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-socket\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-envoy\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-credential-socket\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istiod-ca-cert\") pod \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\" (UID: \"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7\") "
Sep 10 13:46:46.813302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:46.813302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:46.813302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813272    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:46.813562 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813422    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:46.813653 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:46.813653 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813612    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.813653 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813635    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.813653 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813650    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.813869 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.813663    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.815328 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.815288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:46:46.815451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.815333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-kube-api-access-4cjlg" (OuterVolumeSpecName: "kube-api-access-4cjlg") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "kube-api-access-4cjlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:46.815451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.815348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:46.815533 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.815495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" (UID: "da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:46.914908 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.914884    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.914908 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.914907    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjlg\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-kube-api-access-4cjlg\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.915028 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.914916    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.915028 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.914925    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.915028 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:46.914932    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:46:47.229866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.229838    2577 generic.go:356] "Generic (PLEG): container finished" podID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerID="d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494" exitCode=0
Sep 10 13:46:47.230006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.229882    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"
Sep 10 13:46:47.230006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.229902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" event={"ID":"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7","Type":"ContainerDied","Data":"d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494"}
Sep 10 13:46:47.230006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.229946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj" event={"ID":"da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7","Type":"ContainerDied","Data":"1205c3da201de55ff67471c995733c5ff4fd6fbccc45d4a0dde9564dbd963b59"}
Sep 10 13:46:47.230006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.229969    2577 scope.go:117] "RemoveContainer" containerID="d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494"
Sep 10 13:46:47.239782 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.239760    2577 scope.go:117] "RemoveContainer" containerID="2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"
Sep 10 13:46:47.249733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.249716    2577 scope.go:117] "RemoveContainer" containerID="d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494"
Sep 10 13:46:47.250009 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:46:47.249987    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494\": container with ID starting with d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494 not found: ID does not exist" containerID="d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494"
Sep 10 13:46:47.250079 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.250016    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494"} err="failed to get container status \"d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494\": rpc error: code = NotFound desc = could not find container \"d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494\": container with ID starting with d31e0835d46f0ce3650bb62eda7363926fd6b4a309570b9aae97119f7bf52494 not found: ID does not exist"
Sep 10 13:46:47.250079 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.250035    2577 scope.go:117] "RemoveContainer" containerID="2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"
Sep 10 13:46:47.250334 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:46:47.250315    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff\": container with ID starting with 2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff not found: ID does not exist" containerID="2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"
Sep 10 13:46:47.250394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.250338    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff"} err="failed to get container status \"2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff\": rpc error: code = NotFound desc = could not find container \"2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff\": container with ID starting with 2396b6c83164b187c15a2fc16145b9012fbd5a3477ef966d0af0b3e6fe0c7cff not found: ID does not exist"
Sep 10 13:46:47.285619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.285592    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"]
Sep 10 13:46:47.292074 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:47.292053    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ba18dc-openshift-default-8998bc88-wrxcj"]
Sep 10 13:46:48.452348 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.452302    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" path="/var/lib/kubelet/pods/da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7/volumes"
Sep 10 13:46:48.500222 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.500174    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"]
Sep 10 13:46:48.501184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.501156    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy"
Sep 10 13:46:48.501184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.501184    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy"
Sep 10 13:46:48.501436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.501422    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy"
Sep 10 13:46:48.501496 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.501442    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy"
Sep 10 13:46:48.507437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.507405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.510823 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.510795    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"]
Sep 10 13:46:48.511377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.511358    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7eb0a3-serving-cert\""
Sep 10 13:46:48.633101 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.633065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2f7\" (UniqueName: \"kubernetes.io/projected/60f48b11-0d3c-4745-bf49-82c6724acfab-kube-api-access-dz2f7\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.633101 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.633102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f48b11-0d3c-4745-bf49-82c6724acfab-openshift-service-ca\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.633320 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.633183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-maas-api-tls\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.633320 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.633277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-metrics-certs\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.733816 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.733743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-metrics-certs\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.733944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.733822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2f7\" (UniqueName: \"kubernetes.io/projected/60f48b11-0d3c-4745-bf49-82c6724acfab-kube-api-access-dz2f7\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.733944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.733853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f48b11-0d3c-4745-bf49-82c6724acfab-openshift-service-ca\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.733944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.733923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-maas-api-tls\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.734515 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.734491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f48b11-0d3c-4745-bf49-82c6724acfab-openshift-service-ca\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.736354 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.736332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-maas-api-tls\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.736449 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.736366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-metrics-certs\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.748176 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.748152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2f7\" (UniqueName: \"kubernetes.io/projected/60f48b11-0d3c-4745-bf49-82c6724acfab-kube-api-access-dz2f7\") pod \"maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:48.819487 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:48.819463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:49.109380 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:49.109352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"]
Sep 10 13:46:49.110768 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:46:49.110739    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f48b11_0d3c_4745_bf49_82c6724acfab.slice/crio-6b1f673a26603073e3cc1fa17e0fcede9a4cfedb6b41afa87b1e9ef12beabe6d WatchSource:0}: Error finding container 6b1f673a26603073e3cc1fa17e0fcede9a4cfedb6b41afa87b1e9ef12beabe6d: Status 404 returned error can't find the container with id 6b1f673a26603073e3cc1fa17e0fcede9a4cfedb6b41afa87b1e9ef12beabe6d
Sep 10 13:46:49.112302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:49.112284    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:46:49.242463 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:49.242425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6" event={"ID":"60f48b11-0d3c-4745-bf49-82c6724acfab","Type":"ContainerStarted","Data":"6b1f673a26603073e3cc1fa17e0fcede9a4cfedb6b41afa87b1e9ef12beabe6d"}
Sep 10 13:46:50.249445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:50.249395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6" event={"ID":"60f48b11-0d3c-4745-bf49-82c6724acfab","Type":"ContainerStarted","Data":"4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3"}
Sep 10 13:46:50.249935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:50.249469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:46:50.275808 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:50.275754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6" podStartSLOduration=1.7471701020000001 podStartE2EDuration="2.275733225s" totalImagesPullingTime="528.563123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:50.27426176 +0000 UTC m=+954.299076464" watchObservedRunningTime="2026-09-10 13:46:50.275733225 +0000 UTC m=+954.300547930"
Sep 10 13:46:55.006059 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:55.006027    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"]
Sep 10 13:46:56.261422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:46:56.261396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:47:10.027076 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.027010    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7" podUID="e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" containerName="main" containerID="cri-o://262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2" gracePeriod=45
Sep 10 13:47:10.269031 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.269003    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:47:10.362204 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.362177    2577 generic.go:356] "Generic (PLEG): container finished" podID="e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" containerID="262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2" exitCode=0
Sep 10 13:47:10.362361 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.362223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"
Sep 10 13:47:10.362361 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.362277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7" event={"ID":"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f","Type":"ContainerDied","Data":"262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2"}
Sep 10 13:47:10.362361 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.362326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7" event={"ID":"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f","Type":"ContainerDied","Data":"4b52f5526dba1e4eb87feea08602e6fec77030995b113c11eced457339a5e83f"}
Sep 10 13:47:10.362361 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.362350    2577 scope.go:117] "RemoveContainer" containerID="262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2"
Sep 10 13:47:10.370657 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.370642    2577 scope.go:117] "RemoveContainer" containerID="262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2"
Sep 10 13:47:10.370933 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:47:10.370915    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2\": container with ID starting with 262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2 not found: ID does not exist" containerID="262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2"
Sep 10 13:47:10.370976 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.370943    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2"} err="failed to get container status \"262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2\": rpc error: code = NotFound desc = could not find container \"262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2\": container with ID starting with 262ef6b34e7fd8859632056c6147ec961cd182d5658b5e12d57fa8f4d6b079d2 not found: ID does not exist"
Sep 10 13:47:10.416270 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-model-cache\") pod \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") "
Sep 10 13:47:10.416365 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-dshm\") pod \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") "
Sep 10 13:47:10.416419 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tls-certs\") pod \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") "
Sep 10 13:47:10.416419 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416394    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-home\") pod \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") "
Sep 10 13:47:10.416525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxvzm\" (UniqueName: \"kubernetes.io/projected/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-kube-api-access-lxvzm\") pod \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") "
Sep 10 13:47:10.416525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tmp-dir\") pod \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\" (UID: \"e21a39c1-d4f6-489e-bcf2-65d5b8756b6f\") "
Sep 10 13:47:10.416525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" (UID: "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:47:10.416790 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416735    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-home" (OuterVolumeSpecName: "home") pod "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" (UID: "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:47:10.416913 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" (UID: "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:47:10.416977 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416928    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-home\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:47:10.416977 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416948    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tmp-dir\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:47:10.416977 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.416962    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-model-cache\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:47:10.418525 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.418500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-dshm" (OuterVolumeSpecName: "dshm") pod "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" (UID: "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:47:10.418650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.418532    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" (UID: "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:10.418650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.418556    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-kube-api-access-lxvzm" (OuterVolumeSpecName: "kube-api-access-lxvzm") pod "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" (UID: "e21a39c1-d4f6-489e-bcf2-65d5b8756b6f"). InnerVolumeSpecName "kube-api-access-lxvzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:47:10.518392 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.518370    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-tls-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:47:10.518392 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.518391    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvzm\" (UniqueName: \"kubernetes.io/projected/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-kube-api-access-lxvzm\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:47:10.518524 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.518400    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f-dshm\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:47:10.680022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.679956    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"]
Sep 10 13:47:10.684881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:10.684859    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-3696db05-kserve-54b96b7498-htqq7"]
Sep 10 13:47:12.454033 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:12.453999    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" path="/var/lib/kubelet/pods/e21a39c1-d4f6-489e-bcf2-65d5b8756b6f/volumes"
Sep 10 13:47:41.540374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.540303    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"]
Sep 10 13:47:41.540764 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.540749    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" containerName="main"
Sep 10 13:47:41.540764 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.540761    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" containerName="main"
Sep 10 13:47:41.540841 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.540773    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy"
Sep 10 13:47:41.540841 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.540779    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3eaa3a-c4d4-4d85-8f3b-d8cb96d8e1e7" containerName="istio-proxy"
Sep 10 13:47:41.540904 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.540845    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21a39c1-d4f6-489e-bcf2-65d5b8756b6f" containerName="main"
Sep 10 13:47:41.544103 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.544087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.547757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.547730    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7eb0a3\"/\"explicit-model-7eb0a3-kserve-self-signed-certs\""
Sep 10 13:47:41.548697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.548672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7eb0a3\"/\"openshift-service-ca.crt\""
Sep 10 13:47:41.548869 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.548744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7eb0a3\"/\"kube-root-ca.crt\""
Sep 10 13:47:41.548948 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.548761    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7eb0a3\"/\"default-dockercfg-ccd69\""
Sep 10 13:47:41.564110 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.564088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"]
Sep 10 13:47:41.689045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.689014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcaa94-0b75-4921-8236-936918c799ea-tls-certs\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.689045 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.689045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-tmp-dir\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.689230 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.689093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9jp\" (UniqueName: \"kubernetes.io/projected/16dcaa94-0b75-4921-8236-936918c799ea-kube-api-access-jx9jp\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.689230 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.689124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-dshm\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.689230 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.689143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-model-cache\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.689343 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.689233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-home\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.789918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.789890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcaa94-0b75-4921-8236-936918c799ea-tls-certs\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.789930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-tmp-dir\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790046 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.789991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9jp\" (UniqueName: \"kubernetes.io/projected/16dcaa94-0b75-4921-8236-936918c799ea-kube-api-access-jx9jp\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790141 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.790119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-dshm\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790186 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.790163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-model-cache\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790226 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.790199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-home\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790447 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.790399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-tmp-dir\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790542 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.790487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-model-cache\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.790640 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.790611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-home\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.792242 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.792219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-dshm\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.792540 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.792521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcaa94-0b75-4921-8236-936918c799ea-tls-certs\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.800981 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.800960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9jp\" (UniqueName: \"kubernetes.io/projected/16dcaa94-0b75-4921-8236-936918c799ea-kube-api-access-jx9jp\") pod \"explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") " pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:41.854010 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:41.853989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:42.140608 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:47:42.140555    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16dcaa94_0b75_4921_8236_936918c799ea.slice/crio-c36ab7265b2d902fc194eefdfb78fec8595c5e0b124aede2ecf297647dcabe50 WatchSource:0}: Error finding container c36ab7265b2d902fc194eefdfb78fec8595c5e0b124aede2ecf297647dcabe50: Status 404 returned error can't find the container with id c36ab7265b2d902fc194eefdfb78fec8595c5e0b124aede2ecf297647dcabe50
Sep 10 13:47:42.141108 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:42.141085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"]
Sep 10 13:47:42.523758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:42.523668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl" event={"ID":"16dcaa94-0b75-4921-8236-936918c799ea","Type":"ContainerStarted","Data":"e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03"}
Sep 10 13:47:42.523758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:42.523715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl" event={"ID":"16dcaa94-0b75-4921-8236-936918c799ea","Type":"ContainerStarted","Data":"c36ab7265b2d902fc194eefdfb78fec8595c5e0b124aede2ecf297647dcabe50"}
Sep 10 13:47:42.558398 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:42.558355    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl" podStartSLOduration=1.5583411759999999 podStartE2EDuration="1.558341176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:42.55791639 +0000 UTC m=+1006.582731093" watchObservedRunningTime="2026-09-10 13:47:42.558341176 +0000 UTC m=+1006.583155878"
Sep 10 13:47:51.854827 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:51.854796    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:51.855248 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:51.854842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:51.867384 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:51.867360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:52.584006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:52.583981    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:47:54.893909 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:47:54.893875    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"]
Sep 10 13:48:02.837168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.837133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"]
Sep 10 13:48:02.842049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.842030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:02.846498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.846481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-21316764-serving-cert\""
Sep 10 13:48:02.880622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.880600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"]
Sep 10 13:48:02.970164 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.970130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-metrics-certs\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:02.970290 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.970179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153b78f-e208-4a9f-a6de-fa122f891154-openshift-service-ca\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:02.970338 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.970303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-maas-api-tls\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:02.970400 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:02.970385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgf9\" (UniqueName: \"kubernetes.io/projected/a153b78f-e208-4a9f-a6de-fa122f891154-kube-api-access-bqgf9\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.071264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.071234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-metrics-certs\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.071399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.071273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153b78f-e208-4a9f-a6de-fa122f891154-openshift-service-ca\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.071399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.071317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-maas-api-tls\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.071399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.071357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgf9\" (UniqueName: \"kubernetes.io/projected/a153b78f-e208-4a9f-a6de-fa122f891154-kube-api-access-bqgf9\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.072081 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.072060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153b78f-e208-4a9f-a6de-fa122f891154-openshift-service-ca\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.073790 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.073768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-maas-api-tls\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.073871 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.073799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-metrics-certs\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.089042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.088986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgf9\" (UniqueName: \"kubernetes.io/projected/a153b78f-e208-4a9f-a6de-fa122f891154-kube-api-access-bqgf9\") pod \"maas-api-e2e-mt-21316764-5cd685494d-7lf2h\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.152479 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.152453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:03.455906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.455882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"]
Sep 10 13:48:03.458488 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:48:03.458461    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda153b78f_e208_4a9f_a6de_fa122f891154.slice/crio-15c18373a789545f0b33d1d6705651c8f4cec86a2356ffc273e40920269691cd WatchSource:0}: Error finding container 15c18373a789545f0b33d1d6705651c8f4cec86a2356ffc273e40920269691cd: Status 404 returned error can't find the container with id 15c18373a789545f0b33d1d6705651c8f4cec86a2356ffc273e40920269691cd
Sep 10 13:48:03.634232 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:03.634196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h" event={"ID":"a153b78f-e208-4a9f-a6de-fa122f891154","Type":"ContainerStarted","Data":"15c18373a789545f0b33d1d6705651c8f4cec86a2356ffc273e40920269691cd"}
Sep 10 13:48:04.642077 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:04.642043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h" event={"ID":"a153b78f-e208-4a9f-a6de-fa122f891154","Type":"ContainerStarted","Data":"abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f"}
Sep 10 13:48:04.642431 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:04.642149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:04.683094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:04.683029    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h" podStartSLOduration=2.143649861 podStartE2EDuration="2.6830084s" totalImagesPullingTime="539.358539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:04.680742973 +0000 UTC m=+1028.705557677" watchObservedRunningTime="2026-09-10 13:48:04.6830084 +0000 UTC m=+1028.707823104"
Sep 10 13:48:06.835289 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:06.835257    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"]
Sep 10 13:48:06.835685 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:06.835483    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6" podUID="60f48b11-0d3c-4745-bf49-82c6724acfab" containerName="maas-api" containerID="cri-o://4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3" gracePeriod=30
Sep 10 13:48:07.096489 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.096434    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:48:07.211695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.211625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-metrics-certs\") pod \"60f48b11-0d3c-4745-bf49-82c6724acfab\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") "
Sep 10 13:48:07.211695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.211686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz2f7\" (UniqueName: \"kubernetes.io/projected/60f48b11-0d3c-4745-bf49-82c6724acfab-kube-api-access-dz2f7\") pod \"60f48b11-0d3c-4745-bf49-82c6724acfab\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") "
Sep 10 13:48:07.211935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.211781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f48b11-0d3c-4745-bf49-82c6724acfab-openshift-service-ca\") pod \"60f48b11-0d3c-4745-bf49-82c6724acfab\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") "
Sep 10 13:48:07.211935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.211813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-maas-api-tls\") pod \"60f48b11-0d3c-4745-bf49-82c6724acfab\" (UID: \"60f48b11-0d3c-4745-bf49-82c6724acfab\") "
Sep 10 13:48:07.212211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.212186    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f48b11-0d3c-4745-bf49-82c6724acfab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60f48b11-0d3c-4745-bf49-82c6724acfab" (UID: "60f48b11-0d3c-4745-bf49-82c6724acfab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:48:07.213844 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.213818    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60f48b11-0d3c-4745-bf49-82c6724acfab" (UID: "60f48b11-0d3c-4745-bf49-82c6724acfab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:07.213944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.213844    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f48b11-0d3c-4745-bf49-82c6724acfab-kube-api-access-dz2f7" (OuterVolumeSpecName: "kube-api-access-dz2f7") pod "60f48b11-0d3c-4745-bf49-82c6724acfab" (UID: "60f48b11-0d3c-4745-bf49-82c6724acfab"). InnerVolumeSpecName "kube-api-access-dz2f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:07.213944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.213891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60f48b11-0d3c-4745-bf49-82c6724acfab" (UID: "60f48b11-0d3c-4745-bf49-82c6724acfab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:07.313425 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.313393    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2f7\" (UniqueName: \"kubernetes.io/projected/60f48b11-0d3c-4745-bf49-82c6724acfab-kube-api-access-dz2f7\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:07.313425 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.313423    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f48b11-0d3c-4745-bf49-82c6724acfab-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:07.313606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.313437    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:07.313606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.313450    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f48b11-0d3c-4745-bf49-82c6724acfab-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:07.659014 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.658978    2577 generic.go:356] "Generic (PLEG): container finished" podID="60f48b11-0d3c-4745-bf49-82c6724acfab" containerID="4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3" exitCode=0
Sep 10 13:48:07.659200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.659024    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"
Sep 10 13:48:07.659200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.659064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6" event={"ID":"60f48b11-0d3c-4745-bf49-82c6724acfab","Type":"ContainerDied","Data":"4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3"}
Sep 10 13:48:07.659200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.659126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6" event={"ID":"60f48b11-0d3c-4745-bf49-82c6724acfab","Type":"ContainerDied","Data":"6b1f673a26603073e3cc1fa17e0fcede9a4cfedb6b41afa87b1e9ef12beabe6d"}
Sep 10 13:48:07.659200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.659153    2577 scope.go:117] "RemoveContainer" containerID="4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3"
Sep 10 13:48:07.668803 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.668786    2577 scope.go:117] "RemoveContainer" containerID="4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3"
Sep 10 13:48:07.669068 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:48:07.669050    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3\": container with ID starting with 4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3 not found: ID does not exist" containerID="4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3"
Sep 10 13:48:07.669145 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.669075    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3"} err="failed to get container status \"4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3\": rpc error: code = NotFound desc = could not find container \"4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3\": container with ID starting with 4ac215fedebca5d0082db3b2c369745b68a6c23a31e442901a32e6b36fb155d3 not found: ID does not exist"
Sep 10 13:48:07.689075 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.689055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"]
Sep 10 13:48:07.701362 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:07.701341    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7eb0a3-69d498cbbc-khpw6"]
Sep 10 13:48:08.453643 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:08.453610    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f48b11-0d3c-4745-bf49-82c6724acfab" path="/var/lib/kubelet/pods/60f48b11-0d3c-4745-bf49-82c6724acfab/volumes"
Sep 10 13:48:09.913613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:09.913559    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl" podUID="16dcaa94-0b75-4921-8236-936918c799ea" containerName="main" containerID="cri-o://e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03" gracePeriod=45
Sep 10 13:48:10.174193 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.174142    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:48:10.237140 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-home\") pod \"16dcaa94-0b75-4921-8236-936918c799ea\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") "
Sep 10 13:48:10.237278 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-model-cache\") pod \"16dcaa94-0b75-4921-8236-936918c799ea\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") "
Sep 10 13:48:10.237278 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237194    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx9jp\" (UniqueName: \"kubernetes.io/projected/16dcaa94-0b75-4921-8236-936918c799ea-kube-api-access-jx9jp\") pod \"16dcaa94-0b75-4921-8236-936918c799ea\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") "
Sep 10 13:48:10.237278 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-tmp-dir\") pod \"16dcaa94-0b75-4921-8236-936918c799ea\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") "
Sep 10 13:48:10.237278 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcaa94-0b75-4921-8236-936918c799ea-tls-certs\") pod \"16dcaa94-0b75-4921-8236-936918c799ea\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") "
Sep 10 13:48:10.237476 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-home" (OuterVolumeSpecName: "home") pod "16dcaa94-0b75-4921-8236-936918c799ea" (UID: "16dcaa94-0b75-4921-8236-936918c799ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:10.237529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "16dcaa94-0b75-4921-8236-936918c799ea" (UID: "16dcaa94-0b75-4921-8236-936918c799ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:10.237529 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16dcaa94-0b75-4921-8236-936918c799ea" (UID: "16dcaa94-0b75-4921-8236-936918c799ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:10.237622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.237536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-dshm\") pod \"16dcaa94-0b75-4921-8236-936918c799ea\" (UID: \"16dcaa94-0b75-4921-8236-936918c799ea\") "
Sep 10 13:48:10.239520 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.238354    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-home\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:10.239520 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.238379    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-model-cache\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:10.239520 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.238403    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-tmp-dir\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:10.248221 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.248196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-dshm" (OuterVolumeSpecName: "dshm") pod "16dcaa94-0b75-4921-8236-936918c799ea" (UID: "16dcaa94-0b75-4921-8236-936918c799ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:10.248422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.248396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16dcaa94-0b75-4921-8236-936918c799ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16dcaa94-0b75-4921-8236-936918c799ea" (UID: "16dcaa94-0b75-4921-8236-936918c799ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:10.248422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.248408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dcaa94-0b75-4921-8236-936918c799ea-kube-api-access-jx9jp" (OuterVolumeSpecName: "kube-api-access-jx9jp") pod "16dcaa94-0b75-4921-8236-936918c799ea" (UID: "16dcaa94-0b75-4921-8236-936918c799ea"). InnerVolumeSpecName "kube-api-access-jx9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:10.339160 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.339135    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcaa94-0b75-4921-8236-936918c799ea-dshm\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:10.339250 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.339174    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx9jp\" (UniqueName: \"kubernetes.io/projected/16dcaa94-0b75-4921-8236-936918c799ea-kube-api-access-jx9jp\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:10.339250 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.339184    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcaa94-0b75-4921-8236-936918c799ea-tls-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:10.652124 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.652098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:10.677588 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.677550    2577 generic.go:356] "Generic (PLEG): container finished" podID="16dcaa94-0b75-4921-8236-936918c799ea" containerID="e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03" exitCode=0
Sep 10 13:48:10.677723 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.677617    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"
Sep 10 13:48:10.677723 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.677643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl" event={"ID":"16dcaa94-0b75-4921-8236-936918c799ea","Type":"ContainerDied","Data":"e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03"}
Sep 10 13:48:10.677723 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.677708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl" event={"ID":"16dcaa94-0b75-4921-8236-936918c799ea","Type":"ContainerDied","Data":"c36ab7265b2d902fc194eefdfb78fec8595c5e0b124aede2ecf297647dcabe50"}
Sep 10 13:48:10.677870 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.677731    2577 scope.go:117] "RemoveContainer" containerID="e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03"
Sep 10 13:48:10.686658 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.686643    2577 scope.go:117] "RemoveContainer" containerID="e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03"
Sep 10 13:48:10.686952 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:48:10.686903    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03\": container with ID starting with e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03 not found: ID does not exist" containerID="e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03"
Sep 10 13:48:10.686952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.686931    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03"} err="failed to get container status \"e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03\": rpc error: code = NotFound desc = could not find container \"e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03\": container with ID starting with e06205b50baa3718be0c0e7e6fe26ea46def521029b931f1c6d5e3c2445aae03 not found: ID does not exist"
Sep 10 13:48:10.710840 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.710779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"]
Sep 10 13:48:10.715190 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:10.715166    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7eb0a3/explicit-model-7eb0a3-kserve-6b5db9dd9-z2xtl"]
Sep 10 13:48:12.454758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:12.454725    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16dcaa94-0b75-4921-8236-936918c799ea" path="/var/lib/kubelet/pods/16dcaa94-0b75-4921-8236-936918c799ea/volumes"
Sep 10 13:48:16.400622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.400589    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"]
Sep 10 13:48:16.401435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.401412    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f48b11-0d3c-4745-bf49-82c6724acfab" containerName="maas-api"
Sep 10 13:48:16.401500 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.401442    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f48b11-0d3c-4745-bf49-82c6724acfab" containerName="maas-api"
Sep 10 13:48:16.401500 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.401481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16dcaa94-0b75-4921-8236-936918c799ea" containerName="main"
Sep 10 13:48:16.401500 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.401495    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16dcaa94-0b75-4921-8236-936918c799ea" containerName="main"
Sep 10 13:48:16.401670 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.401656    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16dcaa94-0b75-4921-8236-936918c799ea" containerName="main"
Sep 10 13:48:16.401715 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.401677    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f48b11-0d3c-4745-bf49-82c6724acfab" containerName="maas-api"
Sep 10 13:48:16.411055 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.409947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.413906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.413879    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-33cc41-openshift-default-dockercfg-khqq8\""
Sep 10 13:48:16.416378 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.416354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"]
Sep 10 13:48:16.493030 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-envoy\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istiod-ca-cert\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-socket\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbj5p\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-kube-api-access-mbj5p\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493339 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-data\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493339 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-credential-socket\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493410 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-token\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493445 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-podinfo\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.493478 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.493463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-certs\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.594871 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.594844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-envoy\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.594996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.594880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istiod-ca-cert\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.594996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.594915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-socket\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.594996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.594932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbj5p\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-kube-api-access-mbj5p\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.594996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.594957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-data\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595201 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.594994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-credential-socket\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595201 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-token\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595201 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-podinfo\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595201 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-certs\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595398 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-data\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595460 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-certs\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595526 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-socket\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-credential-socket\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.595854 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.595743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istiod-ca-cert\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.597165 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.597142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-envoy\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.597443 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.597425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-podinfo\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.610225 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.610199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbj5p\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-kube-api-access-mbj5p\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.610716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.610697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-token\") pod \"e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") " pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:16.725658 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:16.725562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:17.006077 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.006040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"]
Sep 10 13:48:17.007536 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:48:17.007514    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b87cd9c_0d0b_453e_8dcc_b1524b52ba75.slice/crio-da4c06513f8e6d908480f971505d3fa27be019bd1f5e4b2ef8202fd0b1769ec2 WatchSource:0}: Error finding container da4c06513f8e6d908480f971505d3fa27be019bd1f5e4b2ef8202fd0b1769ec2: Status 404 returned error can't find the container with id da4c06513f8e6d908480f971505d3fa27be019bd1f5e4b2ef8202fd0b1769ec2
Sep 10 13:48:17.009716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.009677    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:17.009784 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.009752    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:17.009821 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.009784    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:17.716834 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.716791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8" event={"ID":"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75","Type":"ContainerStarted","Data":"cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7"}
Sep 10 13:48:17.717295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.716853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8" event={"ID":"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75","Type":"ContainerStarted","Data":"da4c06513f8e6d908480f971505d3fa27be019bd1f5e4b2ef8202fd0b1769ec2"}
Sep 10 13:48:17.726157 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.726128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:17.756623 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:17.756548    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8" podStartSLOduration=1.7565297850000001 podStartE2EDuration="1.756529785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:17.75470489 +0000 UTC m=+1041.779519594" watchObservedRunningTime="2026-09-10 13:48:17.756529785 +0000 UTC m=+1041.781344487"
Sep 10 13:48:18.731223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:18.731194    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:19.727718 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:19.727683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:19.728830 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:19.728810    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:46.958507 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:46.958478    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"]
Sep 10 13:48:46.959003 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:46.958717    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h" podUID="a153b78f-e208-4a9f-a6de-fa122f891154" containerName="maas-api" containerID="cri-o://abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f" gracePeriod=30
Sep 10 13:48:47.212217 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.212160    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:47.370751 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.370720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-metrics-certs\") pod \"a153b78f-e208-4a9f-a6de-fa122f891154\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") "
Sep 10 13:48:47.370905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.370762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqgf9\" (UniqueName: \"kubernetes.io/projected/a153b78f-e208-4a9f-a6de-fa122f891154-kube-api-access-bqgf9\") pod \"a153b78f-e208-4a9f-a6de-fa122f891154\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") "
Sep 10 13:48:47.370905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.370815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153b78f-e208-4a9f-a6de-fa122f891154-openshift-service-ca\") pod \"a153b78f-e208-4a9f-a6de-fa122f891154\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") "
Sep 10 13:48:47.370905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.370848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-maas-api-tls\") pod \"a153b78f-e208-4a9f-a6de-fa122f891154\" (UID: \"a153b78f-e208-4a9f-a6de-fa122f891154\") "
Sep 10 13:48:47.371238 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.371206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a153b78f-e208-4a9f-a6de-fa122f891154-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a153b78f-e208-4a9f-a6de-fa122f891154" (UID: "a153b78f-e208-4a9f-a6de-fa122f891154"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:48:47.371366 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.371289    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153b78f-e208-4a9f-a6de-fa122f891154-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:47.372996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.372965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a153b78f-e208-4a9f-a6de-fa122f891154" (UID: "a153b78f-e208-4a9f-a6de-fa122f891154"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:47.373110 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.373015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a153b78f-e208-4a9f-a6de-fa122f891154" (UID: "a153b78f-e208-4a9f-a6de-fa122f891154"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:47.373110 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.373043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a153b78f-e208-4a9f-a6de-fa122f891154-kube-api-access-bqgf9" (OuterVolumeSpecName: "kube-api-access-bqgf9") pod "a153b78f-e208-4a9f-a6de-fa122f891154" (UID: "a153b78f-e208-4a9f-a6de-fa122f891154"). InnerVolumeSpecName "kube-api-access-bqgf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:47.472826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.472745    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:47.472826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.472772    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153b78f-e208-4a9f-a6de-fa122f891154-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:47.472826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.472781    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgf9\" (UniqueName: \"kubernetes.io/projected/a153b78f-e208-4a9f-a6de-fa122f891154-kube-api-access-bqgf9\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:47.875018 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.874980    2577 generic.go:356] "Generic (PLEG): container finished" podID="a153b78f-e208-4a9f-a6de-fa122f891154" containerID="abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f" exitCode=0
Sep 10 13:48:47.875211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.875024    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"
Sep 10 13:48:47.875211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.875075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h" event={"ID":"a153b78f-e208-4a9f-a6de-fa122f891154","Type":"ContainerDied","Data":"abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f"}
Sep 10 13:48:47.875211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.875126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h" event={"ID":"a153b78f-e208-4a9f-a6de-fa122f891154","Type":"ContainerDied","Data":"15c18373a789545f0b33d1d6705651c8f4cec86a2356ffc273e40920269691cd"}
Sep 10 13:48:47.875211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.875150    2577 scope.go:117] "RemoveContainer" containerID="abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f"
Sep 10 13:48:47.884987 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.884969    2577 scope.go:117] "RemoveContainer" containerID="abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f"
Sep 10 13:48:47.885255 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:48:47.885236    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f\": container with ID starting with abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f not found: ID does not exist" containerID="abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f"
Sep 10 13:48:47.885318 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.885265    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f"} err="failed to get container status \"abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f\": rpc error: code = NotFound desc = could not find container \"abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f\": container with ID starting with abbb4b910b773400912baecac1a7a62cb93b23e0a09ad5b0f712dccbf8b9e01f not found: ID does not exist"
Sep 10 13:48:47.899383 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.899359    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"]
Sep 10 13:48:47.902879 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:47.902862    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21316764-5cd685494d-7lf2h"]
Sep 10 13:48:48.452978 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:48.452936    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a153b78f-e208-4a9f-a6de-fa122f891154" path="/var/lib/kubelet/pods/a153b78f-e208-4a9f-a6de-fa122f891154/volumes"
Sep 10 13:48:52.040508 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:52.040422    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"]
Sep 10 13:48:52.040949 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:52.040625    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8" podUID="8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" containerName="istio-proxy" containerID="cri-o://cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7" gracePeriod=30
Sep 10 13:48:57.307418 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.307351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:57.458741 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.458713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-certs\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.458905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.458763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbj5p\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-kube-api-access-mbj5p\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.458905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.458808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-socket\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.458905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.458833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-envoy\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.458905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.458876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-credential-socket\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.459117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.458964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-token\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.459117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-data\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.459117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:57.459117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istiod-ca-cert\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.459117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459083    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-podinfo\") pod \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\" (UID: \"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75\") "
Sep 10 13:48:57.459393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:57.459393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:57.459509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:48:57.459509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:57.459655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459529    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.459655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459548    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.459655 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.459562    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.461131 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.461097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:57.461251 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.461215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-kube-api-access-mbj5p" (OuterVolumeSpecName: "kube-api-access-mbj5p") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "kube-api-access-mbj5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:57.461420 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.461402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:57.461420 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.461410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" (UID: "8b87cd9c-0d0b-453e-8dcc-b1524b52ba75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:48:57.560118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.560051    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.560118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.560075    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.560118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.560084    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.560118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.560092    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbj5p\" (UniqueName: \"kubernetes.io/projected/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-kube-api-access-mbj5p\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.560118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.560103    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.560118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.560111    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:48:57.931864 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.931828    2577 generic.go:356] "Generic (PLEG): container finished" podID="8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" containerID="cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7" exitCode=0
Sep 10 13:48:57.932022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.931881    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"
Sep 10 13:48:57.932022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.931916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8" event={"ID":"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75","Type":"ContainerDied","Data":"cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7"}
Sep 10 13:48:57.932022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.931969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8" event={"ID":"8b87cd9c-0d0b-453e-8dcc-b1524b52ba75","Type":"ContainerDied","Data":"da4c06513f8e6d908480f971505d3fa27be019bd1f5e4b2ef8202fd0b1769ec2"}
Sep 10 13:48:57.932022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.931997    2577 scope.go:117] "RemoveContainer" containerID="cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7"
Sep 10 13:48:57.941607 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.941589    2577 scope.go:117] "RemoveContainer" containerID="cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7"
Sep 10 13:48:57.941884 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:48:57.941865    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7\": container with ID starting with cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7 not found: ID does not exist" containerID="cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7"
Sep 10 13:48:57.941930 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.941895    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7"} err="failed to get container status \"cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7\": rpc error: code = NotFound desc = could not find container \"cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7\": container with ID starting with cd9f22f2cc18674ff491e3b54ab20dc5ba47f6bd95ac417d3a2165e1492ceeb7 not found: ID does not exist"
Sep 10 13:48:57.959037 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.959017    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"]
Sep 10 13:48:57.974668 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:57.974642    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-33cc41-openshift-default-5897cb6cfd-w2jb8"]
Sep 10 13:48:58.452416 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:48:58.452385    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" path="/var/lib/kubelet/pods/8b87cd9c-0d0b-453e-8dcc-b1524b52ba75/volumes"
Sep 10 13:49:04.191991 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.191958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"]
Sep 10 13:49:04.192436 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.192200    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5" podUID="bb7d5a9f-69c9-47c9-ad05-92ff062fc400" containerName="maas-api" containerID="cri-o://90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439" gracePeriod=30
Sep 10 13:49:04.446796 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.446739    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:49:04.513769 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.513738    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-maas-api-tls\") pod \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") "
Sep 10 13:49:04.513916 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.513834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-metrics-certs\") pod \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") "
Sep 10 13:49:04.513916 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.513852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-openshift-service-ca\") pod \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") "
Sep 10 13:49:04.513916 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.513882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz8xv\" (UniqueName: \"kubernetes.io/projected/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-kube-api-access-jz8xv\") pod \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\" (UID: \"bb7d5a9f-69c9-47c9-ad05-92ff062fc400\") "
Sep 10 13:49:04.514184 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.514159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb7d5a9f-69c9-47c9-ad05-92ff062fc400" (UID: "bb7d5a9f-69c9-47c9-ad05-92ff062fc400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:04.515937 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.515912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-kube-api-access-jz8xv" (OuterVolumeSpecName: "kube-api-access-jz8xv") pod "bb7d5a9f-69c9-47c9-ad05-92ff062fc400" (UID: "bb7d5a9f-69c9-47c9-ad05-92ff062fc400"). InnerVolumeSpecName "kube-api-access-jz8xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:04.516049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.515938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb7d5a9f-69c9-47c9-ad05-92ff062fc400" (UID: "bb7d5a9f-69c9-47c9-ad05-92ff062fc400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:04.516049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.515957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb7d5a9f-69c9-47c9-ad05-92ff062fc400" (UID: "bb7d5a9f-69c9-47c9-ad05-92ff062fc400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:04.615626 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.615574    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:04.615626 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.615627    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:04.615807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.615637    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:04.615807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.615649    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz8xv\" (UniqueName: \"kubernetes.io/projected/bb7d5a9f-69c9-47c9-ad05-92ff062fc400-kube-api-access-jz8xv\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:04.969380 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.969345    2577 generic.go:356] "Generic (PLEG): container finished" podID="bb7d5a9f-69c9-47c9-ad05-92ff062fc400" containerID="90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439" exitCode=0
Sep 10 13:49:04.969543 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.969401    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"
Sep 10 13:49:04.969543 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.969428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5" event={"ID":"bb7d5a9f-69c9-47c9-ad05-92ff062fc400","Type":"ContainerDied","Data":"90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439"}
Sep 10 13:49:04.969543 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.969479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5" event={"ID":"bb7d5a9f-69c9-47c9-ad05-92ff062fc400","Type":"ContainerDied","Data":"f19af41bb98a1083d5b741640d1edb3849fad17fb79a52277d3054fe8d7e9520"}
Sep 10 13:49:04.969543 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.969500    2577 scope.go:117] "RemoveContainer" containerID="90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439"
Sep 10 13:49:04.981124 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.981103    2577 scope.go:117] "RemoveContainer" containerID="90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439"
Sep 10 13:49:04.981610 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:49:04.981567    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439\": container with ID starting with 90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439 not found: ID does not exist" containerID="90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439"
Sep 10 13:49:04.981706 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.981616    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439"} err="failed to get container status \"90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439\": rpc error: code = NotFound desc = could not find container \"90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439\": container with ID starting with 90863293ceaf58e5f590b0c9ec89f75fa985bb86a3f761846b58f3be0f17a439 not found: ID does not exist"
Sep 10 13:49:04.995882 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:04.995862    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"]
Sep 10 13:49:05.002146 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:05.002118    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f1d168-558d5ff5d8-xzfm5"]
Sep 10 13:49:06.458846 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:06.458811    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7d5a9f-69c9-47c9-ad05-92ff062fc400" path="/var/lib/kubelet/pods/bb7d5a9f-69c9-47c9-ad05-92ff062fc400/volumes"
Sep 10 13:49:15.433469 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.433434    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"]
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.433964    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" containerName="istio-proxy"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.433979    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" containerName="istio-proxy"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.433988    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a153b78f-e208-4a9f-a6de-fa122f891154" containerName="maas-api"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.433993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a153b78f-e208-4a9f-a6de-fa122f891154" containerName="maas-api"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.434008    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7d5a9f-69c9-47c9-ad05-92ff062fc400" containerName="maas-api"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.434014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7d5a9f-69c9-47c9-ad05-92ff062fc400" containerName="maas-api"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.434106    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a153b78f-e208-4a9f-a6de-fa122f891154" containerName="maas-api"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.434114    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b87cd9c-0d0b-453e-8dcc-b1524b52ba75" containerName="istio-proxy"
Sep 10 13:49:15.435996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.434124    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7d5a9f-69c9-47c9-ad05-92ff062fc400" containerName="maas-api"
Sep 10 13:49:15.437178 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.437158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.439991 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.439970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-3d0caa-openshift-default-dockercfg-872qm\""
Sep 10 13:49:15.453654 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.453630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"]
Sep 10 13:49:15.512724 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-envoy\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512828 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-socket\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512828 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-certs\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512828 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-credential-socket\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-podinfo\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-token\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshg9\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-kube-api-access-kshg9\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.512961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-data\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.513125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.512983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istiod-ca-cert\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614149 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-data\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istiod-ca-cert\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614256 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-envoy\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614343 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-socket\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614343 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-certs\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614343 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-credential-socket\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614465 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-podinfo\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614465 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-token\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614465 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshg9\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-kube-api-access-kshg9\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614606 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-data\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-socket\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614837 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-credential-socket\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.614993 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.614971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istiod-ca-cert\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.615174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.615078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-certs\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.616683 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.616659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-envoy\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.616764 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.616725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-podinfo\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.628049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.628019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-token\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.628398 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.628379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshg9\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-kube-api-access-kshg9\") pod \"e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") " pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:15.748030 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:15.747962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:16.030542 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:16.030520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"]
Sep 10 13:49:16.032433 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:49:16.032404    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be83cce_9d30_43f2_b2f6_a91f6f9e6b3b.slice/crio-864064a7527132102914b3df278756c9ba34883e445983e7ea417681562dc61a WatchSource:0}: Error finding container 864064a7527132102914b3df278756c9ba34883e445983e7ea417681562dc61a: Status 404 returned error can't find the container with id 864064a7527132102914b3df278756c9ba34883e445983e7ea417681562dc61a
Sep 10 13:49:16.034673 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:16.034645    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:49:16.034730 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:16.034705    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:49:16.034770 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:16.034732    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:49:17.034925 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:17.034890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" event={"ID":"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b","Type":"ContainerStarted","Data":"2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"}
Sep 10 13:49:17.035385 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:17.034946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" event={"ID":"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b","Type":"ContainerStarted","Data":"864064a7527132102914b3df278756c9ba34883e445983e7ea417681562dc61a"}
Sep 10 13:49:17.076555 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:17.076508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podStartSLOduration=2.076493852 podStartE2EDuration="2.076493852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:17.075977982 +0000 UTC m=+1101.100792695" watchObservedRunningTime="2026-09-10 13:49:17.076493852 +0000 UTC m=+1101.101308554"
Sep 10 13:49:17.748915 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:17.748878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:17.753527 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:17.753505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:18.040596 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:18.040502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:18.041439 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:18.041414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:20.491322 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.491267    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"]
Sep 10 13:49:20.495502 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.495479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.497800 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.497776    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3d0caa-serving-cert\""
Sep 10 13:49:20.508332 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.508296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"]
Sep 10 13:49:20.559451 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.559430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7r58\" (UniqueName: \"kubernetes.io/projected/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-kube-api-access-t7r58\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.559561 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.559464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.559561 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.559504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.559692 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.559605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.660149 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.660125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.660286 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.660176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.660286 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.660238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.660399 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.660315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7r58\" (UniqueName: \"kubernetes.io/projected/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-kube-api-access-t7r58\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.660753 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.660730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.662516 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.662500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.662625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.662605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.672532 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.672511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7r58\" (UniqueName: \"kubernetes.io/projected/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-kube-api-access-t7r58\") pod \"maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:20.807180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:20.807127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:21.085075 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:21.085054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"]
Sep 10 13:49:21.087394 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:49:21.087364    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9559de0b_ddd6_4ab2_8dce_ee31e32114bc.slice/crio-ff8555068ec3f0ef3678d3f108a5458bed874757290c2855f21fbb3cf783c4bc WatchSource:0}: Error finding container ff8555068ec3f0ef3678d3f108a5458bed874757290c2855f21fbb3cf783c4bc: Status 404 returned error can't find the container with id ff8555068ec3f0ef3678d3f108a5458bed874757290c2855f21fbb3cf783c4bc
Sep 10 13:49:22.065289 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:22.065248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2" event={"ID":"9559de0b-ddd6-4ab2-8dce-ee31e32114bc","Type":"ContainerStarted","Data":"ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0"}
Sep 10 13:49:22.065889 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:22.065863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2" event={"ID":"9559de0b-ddd6-4ab2-8dce-ee31e32114bc","Type":"ContainerStarted","Data":"ff8555068ec3f0ef3678d3f108a5458bed874757290c2855f21fbb3cf783c4bc"}
Sep 10 13:49:22.067068 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:22.067039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:22.097689 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:22.097646    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2" podStartSLOduration=1.754372653 podStartE2EDuration="2.097634662s" totalImagesPullingTime="343.262009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:22.096102145 +0000 UTC m=+1106.120916848" watchObservedRunningTime="2026-09-10 13:49:22.097634662 +0000 UTC m=+1106.122449363"
Sep 10 13:49:23.381262 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.381234    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"]
Sep 10 13:49:23.381668 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.381437    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" podUID="abce2590-c3ea-4603-bb3c-0e8538b8f5c2" containerName="maas-api" containerID="cri-o://8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450" gracePeriod=30
Sep 10 13:49:23.640249 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.640191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:49:23.790055 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.790018    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-metrics-certs\") pod \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") "
Sep 10 13:49:23.790258 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.790080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls\") pod \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") "
Sep 10 13:49:23.790258 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.790117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-openshift-service-ca\") pod \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") "
Sep 10 13:49:23.790258 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.790161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk92f\" (UniqueName: \"kubernetes.io/projected/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-kube-api-access-gk92f\") pod \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\" (UID: \"abce2590-c3ea-4603-bb3c-0e8538b8f5c2\") "
Sep 10 13:49:23.790857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.790822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abce2590-c3ea-4603-bb3c-0e8538b8f5c2" (UID: "abce2590-c3ea-4603-bb3c-0e8538b8f5c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:23.792838 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.792809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "abce2590-c3ea-4603-bb3c-0e8538b8f5c2" (UID: "abce2590-c3ea-4603-bb3c-0e8538b8f5c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:23.792923 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.792834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abce2590-c3ea-4603-bb3c-0e8538b8f5c2" (UID: "abce2590-c3ea-4603-bb3c-0e8538b8f5c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:23.793241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.793201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-kube-api-access-gk92f" (OuterVolumeSpecName: "kube-api-access-gk92f") pod "abce2590-c3ea-4603-bb3c-0e8538b8f5c2" (UID: "abce2590-c3ea-4603-bb3c-0e8538b8f5c2"). InnerVolumeSpecName "kube-api-access-gk92f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:23.891906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.891841    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:23.891906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.891864    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:23.891906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.891873    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:23.891906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:23.891881    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk92f\" (UniqueName: \"kubernetes.io/projected/abce2590-c3ea-4603-bb3c-0e8538b8f5c2-kube-api-access-gk92f\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:49:24.076388 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.076350    2577 generic.go:356] "Generic (PLEG): container finished" podID="abce2590-c3ea-4603-bb3c-0e8538b8f5c2" containerID="8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450" exitCode=0
Sep 10 13:49:24.076544 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.076402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"
Sep 10 13:49:24.076544 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.076433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" event={"ID":"abce2590-c3ea-4603-bb3c-0e8538b8f5c2","Type":"ContainerDied","Data":"8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450"}
Sep 10 13:49:24.076544 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.076492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg" event={"ID":"abce2590-c3ea-4603-bb3c-0e8538b8f5c2","Type":"ContainerDied","Data":"cf324ed80330252740c8bca23f400044bf77828a87bc777eeba4cd2ee5a826be"}
Sep 10 13:49:24.076544 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.076515    2577 scope.go:117] "RemoveContainer" containerID="8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450"
Sep 10 13:49:24.086402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.086383    2577 scope.go:117] "RemoveContainer" containerID="8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450"
Sep 10 13:49:24.086692 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:49:24.086672    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450\": container with ID starting with 8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450 not found: ID does not exist" containerID="8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450"
Sep 10 13:49:24.086755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.086703    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450"} err="failed to get container status \"8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450\": rpc error: code = NotFound desc = could not find container \"8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450\": container with ID starting with 8797e7283cdbb1236731461979be6152f36c83319568dfbe46140611a0bd9450 not found: ID does not exist"
Sep 10 13:49:24.102044 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.102023    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"]
Sep 10 13:49:24.109992 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.109973    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e6d037-84d8df8c78-5hskg"]
Sep 10 13:49:24.452796 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:24.452769    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abce2590-c3ea-4603-bb3c-0e8538b8f5c2" path="/var/lib/kubelet/pods/abce2590-c3ea-4603-bb3c-0e8538b8f5c2/volumes"
Sep 10 13:49:29.082504 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:29.082476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:49:34.082755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:34.082713    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 13:49:34.083157 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:34.082793    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 13:49:35.137020 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:35.136988    2577 generic.go:356] "Generic (PLEG): container finished" podID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerID="2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57" exitCode=0
Sep 10 13:49:35.137458 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:35.137043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" event={"ID":"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b","Type":"ContainerDied","Data":"2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"}
Sep 10 13:49:35.137458 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:35.137405    2577 scope.go:117] "RemoveContainer" containerID="2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"
Sep 10 13:49:35.138394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:35.138360    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:49:35.138479 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:35.138433    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:49:35.138479 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:35.138462    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:49:36.147806 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:36.147752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" event={"ID":"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b","Type":"ContainerStarted","Data":"501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3"}
Sep 10 13:49:36.748203 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:36.748174    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:36.748460 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:36.748329    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 13:49:36.748638 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:36.748601    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 13:49:37.748903 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:37.748855    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 13:49:37.749393 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:37.748955    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 13:49:38.748566 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:38.748524    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 13:49:38.748764 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:38.748625    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 13:49:39.748974 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:39.748938    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 13:49:39.749522 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:39.749491    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 13:49:40.748794 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:40.748751    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 13:49:40.748975 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:40.748832    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 13:49:41.751763 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:41.751734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:41.752146 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:41.752002    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:41.752674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:41.752656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:49:45.114662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.114630    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"]
Sep 10 13:49:45.115329 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.115305    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abce2590-c3ea-4603-bb3c-0e8538b8f5c2" containerName="maas-api"
Sep 10 13:49:45.115497 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.115482    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abce2590-c3ea-4603-bb3c-0e8538b8f5c2" containerName="maas-api"
Sep 10 13:49:45.115716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.115696    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abce2590-c3ea-4603-bb3c-0e8538b8f5c2" containerName="maas-api"
Sep 10 13:49:45.118034 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.118013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.120397 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.120376    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b3723e-serving-cert\""
Sep 10 13:49:45.134868 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.134843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"]
Sep 10 13:49:45.208677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.208642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqnp\" (UniqueName: \"kubernetes.io/projected/fb82f7d2-acf3-4173-820f-524de10dec90-kube-api-access-dpqnp\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.208830 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.208749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.208830 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.208791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.208944 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.208860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f7d2-acf3-4173-820f-524de10dec90-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.309999 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.309966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqnp\" (UniqueName: \"kubernetes.io/projected/fb82f7d2-acf3-4173-820f-524de10dec90-kube-api-access-dpqnp\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.310168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.310058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.310168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.310100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.310168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.310156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f7d2-acf3-4173-820f-524de10dec90-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.310326 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:49:45.310275    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-serving-cert: secret "maas-api-e2e-shared-b-w2-b3723e-serving-cert" not found
Sep 10 13:49:45.310379 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:49:45.310332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls podName:fb82f7d2-acf3-4173-820f-524de10dec90 nodeName:}" failed. No retries permitted until 2026-09-10 13:49:45.81031765 +0000 UTC m=+1129.835132330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" (UID: "fb82f7d2-acf3-4173-820f-524de10dec90") : secret "maas-api-e2e-shared-b-w2-b3723e-serving-cert" not found
Sep 10 13:49:45.310856 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.310833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f7d2-acf3-4173-820f-524de10dec90-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.312444 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.312422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.323319 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.323293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqnp\" (UniqueName: \"kubernetes.io/projected/fb82f7d2-acf3-4173-820f-524de10dec90-kube-api-access-dpqnp\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.813888 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.813863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:45.816180 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:45.816156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:46.029151 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:46.029075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:46.332887 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:46.332864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"]
Sep 10 13:49:46.335211 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:49:46.335183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb82f7d2_acf3_4173_820f_524de10dec90.slice/crio-d19fb01b806995929ed1717539c8c254e909c6de2c0438024a55515422f68796 WatchSource:0}: Error finding container d19fb01b806995929ed1717539c8c254e909c6de2c0438024a55515422f68796: Status 404 returned error can't find the container with id d19fb01b806995929ed1717539c8c254e909c6de2c0438024a55515422f68796
Sep 10 13:49:47.214935 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:47.214902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" event={"ID":"fb82f7d2-acf3-4173-820f-524de10dec90","Type":"ContainerStarted","Data":"ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05"}
Sep 10 13:49:47.215123 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:47.214959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" event={"ID":"fb82f7d2-acf3-4173-820f-524de10dec90","Type":"ContainerStarted","Data":"d19fb01b806995929ed1717539c8c254e909c6de2c0438024a55515422f68796"}
Sep 10 13:49:47.215123 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:47.215048    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:49:47.287200 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:47.287154    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" podStartSLOduration=1.91959884 podStartE2EDuration="2.287139448s" totalImagesPullingTime="367.540608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:47.270914381 +0000 UTC m=+1131.295729082" watchObservedRunningTime="2026-09-10 13:49:47.287139448 +0000 UTC m=+1131.311954149"
Sep 10 13:49:53.225006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:49:53.224975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:50:06.792347 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.792315    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"]
Sep 10 13:50:06.796203 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.796181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.800067 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.800047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d5c012-openshift-default-dockercfg-w648d\""
Sep 10 13:50:06.832991 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.832964    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"]
Sep 10 13:50:06.891022 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.890992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-certs\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-credential-socket\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvt6h\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-kube-api-access-mvt6h\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-podinfo\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-socket\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-token\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891414 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-data\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891414 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.891414 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.891296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-envoy\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992515 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-certs\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-credential-socket\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvt6h\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-kube-api-access-mvt6h\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-podinfo\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992699 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-socket\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-token\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-data\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.992922 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.992848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-envoy\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.993122 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.993005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-credential-socket\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.993122 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.993077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-socket\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.993241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.993127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-data\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.993241 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.993157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-certs\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.993439 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.993422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.994936 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.994915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-envoy\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:06.995041 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:06.995001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-podinfo\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:07.007498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.007475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-token\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:07.009647 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.009627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvt6h\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-kube-api-access-mvt6h\") pod \"e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") " pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:07.105953 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.105923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:07.393605 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.393557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"]
Sep 10 13:50:07.396196 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:50:07.396168    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d18c6c_53db_4b10_b6a9_9bedf83ec7fd.slice/crio-256aa53f75d6931992e0abf176a0d4f4e3282d68c5e970b2ccf2b209a9fd5da6 WatchSource:0}: Error finding container 256aa53f75d6931992e0abf176a0d4f4e3282d68c5e970b2ccf2b209a9fd5da6: Status 404 returned error can't find the container with id 256aa53f75d6931992e0abf176a0d4f4e3282d68c5e970b2ccf2b209a9fd5da6
Sep 10 13:50:07.398329 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.398296    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:07.398420 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.398375    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:07.398482 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:07.398415    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:08.329799 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:08.329762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" event={"ID":"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd","Type":"ContainerStarted","Data":"ccb5b864721d94c0118c1ea54447389c663d4ae7f64a6dd0c601f9d9f11d8390"}
Sep 10 13:50:08.330235 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:08.329818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" event={"ID":"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd","Type":"ContainerStarted","Data":"256aa53f75d6931992e0abf176a0d4f4e3282d68c5e970b2ccf2b209a9fd5da6"}
Sep 10 13:50:08.359564 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:08.359519    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podStartSLOduration=2.359482777 podStartE2EDuration="2.359482777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:08.358116147 +0000 UTC m=+1152.382930848" watchObservedRunningTime="2026-09-10 13:50:08.359482777 +0000 UTC m=+1152.384297478"
Sep 10 13:50:09.106214 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:09.106182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:09.111089 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:09.111053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:09.336039 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:09.336007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:09.337146 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:09.337124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:13.588234 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.588202    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"]
Sep 10 13:50:13.591150 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.591126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.594335 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.594304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d5c012\""
Sep 10 13:50:13.594335 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.594321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d5c012-dockercfg-d2c9c\""
Sep 10 13:50:13.620231 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.620205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"]
Sep 10 13:50:13.759264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.759243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d066f-8f92-425c-8e89-7151b3837c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.759402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.759369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/371d066f-8f92-425c-8e89-7151b3837c4c-kube-api-access-54ggj\") pod \"payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.860812 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.860754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d066f-8f92-425c-8e89-7151b3837c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.860985 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.860858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/371d066f-8f92-425c-8e89-7151b3837c4c-kube-api-access-54ggj\") pod \"payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.861459 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.861438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d066f-8f92-425c-8e89-7151b3837c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.885622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.885603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/371d066f-8f92-425c-8e89-7151b3837c4c-kube-api-access-54ggj\") pod \"payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:13.902547 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:13.902529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:14.206463 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:14.206439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"]
Sep 10 13:50:14.208230 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:50:14.208205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d066f_8f92_425c_8e89_7151b3837c4c.slice/crio-2c8c52b86471102daea5c931fee6c15d78b519db34cf9280ec9b96caa39c181e WatchSource:0}: Error finding container 2c8c52b86471102daea5c931fee6c15d78b519db34cf9280ec9b96caa39c181e: Status 404 returned error can't find the container with id 2c8c52b86471102daea5c931fee6c15d78b519db34cf9280ec9b96caa39c181e
Sep 10 13:50:14.365706 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:14.365674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6" event={"ID":"371d066f-8f92-425c-8e89-7151b3837c4c","Type":"ContainerStarted","Data":"2c8c52b86471102daea5c931fee6c15d78b519db34cf9280ec9b96caa39c181e"}
Sep 10 13:50:17.388969 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:17.388935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6" event={"ID":"371d066f-8f92-425c-8e89-7151b3837c4c","Type":"ContainerStarted","Data":"fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7"}
Sep 10 13:50:17.389330 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:17.389085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:17.416099 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:17.416050    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6" podStartSLOduration=2.291886177 podStartE2EDuration="4.416035974s" totalImagesPullingTime="2.124149797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:17.413941224 +0000 UTC m=+1161.438755928" watchObservedRunningTime="2026-09-10 13:50:17.416035974 +0000 UTC m=+1161.440850672"
Sep 10 13:50:18.430312 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.430281    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"]
Sep 10 13:50:18.433160 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.433139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.435813 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.435794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a6883ccf-openshift-default-dockercfg-zc7j4\""
Sep 10 13:50:18.453674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.453652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"]
Sep 10 13:50:18.609561 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istiod-ca-cert\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.609561 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-envoy\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.609826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-credential-socket\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.609826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-podinfo\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.609826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-data\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.609826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-token\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.609826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d468t\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-kube-api-access-d468t\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.610006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-certs\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.610006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.609917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-socket\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711002 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.710931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-podinfo\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711002 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.710974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-data\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711002 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.710992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-token\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711277 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d468t\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-kube-api-access-d468t\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711277 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-certs\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711277 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-socket\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711277 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istiod-ca-cert\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711387    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-data\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-certs\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711498 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-envoy\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711679 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-socket\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711679 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-credential-socket\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-credential-socket\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.711863 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.711833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istiod-ca-cert\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.713492 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.713466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-envoy\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.713657 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.713635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-podinfo\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.724512 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.724484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-token\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.724638 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.724570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d468t\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-kube-api-access-d468t\") pod \"e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") " pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:18.743464 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:18.743443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:19.047556 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.047518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"]
Sep 10 13:50:19.050395 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:50:19.050365    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c9f061_9ed9_4b4e_acab_bf1a4a5ef839.slice/crio-c5cf46571153212d3b9435209faca84744789d8af3fae833fc5c309a237a255a WatchSource:0}: Error finding container c5cf46571153212d3b9435209faca84744789d8af3fae833fc5c309a237a255a: Status 404 returned error can't find the container with id c5cf46571153212d3b9435209faca84744789d8af3fae833fc5c309a237a255a
Sep 10 13:50:19.052905 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.052863    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:19.052996 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.052955    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:19.053042 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.052998    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:19.401509 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.401473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" event={"ID":"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839","Type":"ContainerStarted","Data":"27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"}
Sep 10 13:50:19.401708 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.401527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" event={"ID":"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839","Type":"ContainerStarted","Data":"c5cf46571153212d3b9435209faca84744789d8af3fae833fc5c309a237a255a"}
Sep 10 13:50:19.744032 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:19.743953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:20.751798 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:20.751761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:20.778220 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:20.778163    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podStartSLOduration=2.778146546 podStartE2EDuration="2.778146546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:19.438835005 +0000 UTC m=+1163.463649710" watchObservedRunningTime="2026-09-10 13:50:20.778146546 +0000 UTC m=+1164.802961247"
Sep 10 13:50:21.411894 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:21.411857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:21.412702 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:21.412679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:25.377786 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:25.377730    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 13:50:25.378307 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:25.377849    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 13:50:27.449559 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:27.449481    2577 generic.go:356] "Generic (PLEG): container finished" podID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerID="ccb5b864721d94c0118c1ea54447389c663d4ae7f64a6dd0c601f9d9f11d8390" exitCode=0
Sep 10 13:50:27.449919 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:27.449556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" event={"ID":"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd","Type":"ContainerDied","Data":"ccb5b864721d94c0118c1ea54447389c663d4ae7f64a6dd0c601f9d9f11d8390"}
Sep 10 13:50:27.449966 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:27.449954    2577 scope.go:117] "RemoveContainer" containerID="ccb5b864721d94c0118c1ea54447389c663d4ae7f64a6dd0c601f9d9f11d8390"
Sep 10 13:50:27.450951 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:27.450923    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:27.451056 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:27.450987    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:27.451056 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:27.451021    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:28.397119 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:28.397092    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 13:50:28.462712 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:28.462676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" event={"ID":"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd","Type":"ContainerStarted","Data":"0a97b351b09fc953ff6f0cdbbe69ff02a15a362e8434b8bb88a56aebb4537c47"}
Sep 10 13:50:29.106652 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:29.106620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:29.106813 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:29.106774    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:50:29.106859 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:29.106829    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:50:30.106674 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:30.106641    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:50:30.107087 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:30.106694    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:50:31.106426 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:31.106392    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:50:31.106632 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:31.106453    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:50:32.106851 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:32.106813    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:50:32.107216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:32.106880    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:50:33.107232 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:33.107196    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:50:33.107625 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:33.107275    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:50:34.109708 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:34.109680    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:34.110081 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:34.109941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:34.110520 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:34.110501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 13:50:47.585136 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:47.585106    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerID="27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036" exitCode=0
Sep 10 13:50:47.585553 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:47.585178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" event={"ID":"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839","Type":"ContainerDied","Data":"27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"}
Sep 10 13:50:47.585620 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:47.585593    2577 scope.go:117] "RemoveContainer" containerID="27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"
Sep 10 13:50:47.586722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:47.586691    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:47.586798 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:47.586763    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:47.586834 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:47.586795    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:48.592662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:48.592623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" event={"ID":"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839","Type":"ContainerStarted","Data":"e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb"}
Sep 10 13:50:48.743817 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:48.743778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:48.745060 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:48.745025    2577 patch_prober.go:28] interesting pod/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 13:50:48.745193 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:48.745088    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 13:50:49.744090 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:49.744047    2577 patch_prober.go:28] interesting pod/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 13:50:49.744556 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:49.744121    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 13:50:50.744711 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:50.744677    2577 patch_prober.go:28] interesting pod/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 13:50:50.745183 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:50.744783    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 13:50:51.412216 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:51.412175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:51.744866 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:51.744790    2577 patch_prober.go:28] interesting pod/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 13:50:51.745283 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:51.745251    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 13:50:52.744304 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:52.744263    2577 patch_prober.go:28] interesting pod/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 13:50:52.744499 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:52.744346    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 13:50:53.748289 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:53.748256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:53.749309 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:53.749290    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:50:56.497849 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:56.497816    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:50:56.498328 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:50:56.497882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:51:50.888820 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.888749    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"]
Sep 10 13:51:50.892690 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.892668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:50.910868 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.910849    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5c012\"/\"auth-test-model-d5c012-kserve-self-signed-certs\""
Sep 10 13:51:50.911758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.911741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5c012\"/\"kube-root-ca.crt\""
Sep 10 13:51:50.911860 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.911758    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5c012\"/\"openshift-service-ca.crt\""
Sep 10 13:51:50.916843 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.916820    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5c012\"/\"default-dockercfg-5rrv9\""
Sep 10 13:51:50.940806 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.940777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"]
Sep 10 13:51:50.981538 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.981500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-dshm\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:50.981743 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.981639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-tmp-dir\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:50.981743 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.981684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba194fb6-caf7-4519-9a2b-2827245b6649-tls-certs\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:50.981879 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.981802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-home\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:50.981879 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.981848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-model-cache\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:50.981985 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:50.981877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zlq\" (UniqueName: \"kubernetes.io/projected/ba194fb6-caf7-4519-9a2b-2827245b6649-kube-api-access-64zlq\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083334 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-home\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-model-cache\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zlq\" (UniqueName: \"kubernetes.io/projected/ba194fb6-caf7-4519-9a2b-2827245b6649-kube-api-access-64zlq\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-dshm\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-tmp-dir\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083510 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba194fb6-caf7-4519-9a2b-2827245b6649-tls-certs\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083831 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-model-cache\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083891 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-home\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.083891 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.083874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-tmp-dir\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.085895 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.085856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-dshm\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.086048 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.086031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba194fb6-caf7-4519-9a2b-2827245b6649-tls-certs\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.096326 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.096295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zlq\" (UniqueName: \"kubernetes.io/projected/ba194fb6-caf7-4519-9a2b-2827245b6649-kube-api-access-64zlq\") pod \"auth-test-model-d5c012-kserve-6d876dbbbc-zdsng\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") " pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.201787 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.201702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:51:51.483148 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.483121    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"]
Sep 10 13:51:51.485095 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:51:51.485068    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba194fb6_caf7_4519_9a2b_2827245b6649.slice/crio-fe93a462d76df783f57ec9c61d8e9e49a6cc1330d2e2e2590f0ffc17f8dd0227 WatchSource:0}: Error finding container fe93a462d76df783f57ec9c61d8e9e49a6cc1330d2e2e2590f0ffc17f8dd0227: Status 404 returned error can't find the container with id fe93a462d76df783f57ec9c61d8e9e49a6cc1330d2e2e2590f0ffc17f8dd0227
Sep 10 13:51:51.941370 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.941331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng" event={"ID":"ba194fb6-caf7-4519-9a2b-2827245b6649","Type":"ContainerStarted","Data":"52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9"}
Sep 10 13:51:51.941757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.941378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng" event={"ID":"ba194fb6-caf7-4519-9a2b-2827245b6649","Type":"ContainerStarted","Data":"fe93a462d76df783f57ec9c61d8e9e49a6cc1330d2e2e2590f0ffc17f8dd0227"}
Sep 10 13:51:51.962638 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:51:51.962562    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng" podStartSLOduration=1.9625423579999999 podStartE2EDuration="1.962542358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:51.960192947 +0000 UTC m=+1255.985007649" watchObservedRunningTime="2026-09-10 13:51:51.962542358 +0000 UTC m=+1255.987357063"
Sep 10 13:52:01.202790 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:52:01.202752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:52:01.203250 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:52:01.202838    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:52:01.215437 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:52:01.215412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:52:02.008321 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:52:02.008287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:53:52.568884 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:52.568849    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"]
Sep 10 13:53:52.569416 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:52.569100    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" containerID="cri-o://e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb" gracePeriod=30
Sep 10 13:53:53.750760 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:53.750724    2577 patch_prober.go:28] interesting pod/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:53:53.751134 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:53.750783    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:53:57.823251 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.823227    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:53:57.855774 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-podinfo\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.855774 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-token\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.855952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-socket\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.855952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-envoy\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.855952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d468t\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-kube-api-access-d468t\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.855952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-credential-socket\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.855952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-certs\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.856209 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855958    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istiod-ca-cert\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.856209 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.855984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-data\") pod \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\" (UID: \"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839\") "
Sep 10 13:53:57.856209 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856143    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:53:57.856370 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:53:57.856483 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:53:57.856596 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856544    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:53:57.856677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856594    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.856677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856618    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.856677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856633    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.856830 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.856758    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:53:57.858090 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.858065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-kube-api-access-d468t" (OuterVolumeSpecName: "kube-api-access-d468t") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "kube-api-access-d468t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:53:57.858352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.858327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:53:57.858352 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.858341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:53:57.858677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.858659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" (UID: "d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:53:57.957215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.957185    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.957215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.957209    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.957215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.957219    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.957402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.957227    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d468t\" (UniqueName: \"kubernetes.io/projected/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-kube-api-access-d468t\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.957402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.957237    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:57.957402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:57.957245    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:53:58.645053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.645015    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerID="e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb" exitCode=0
Sep 10 13:53:58.645205 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.645058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"
Sep 10 13:53:58.645205 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.645101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" event={"ID":"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839","Type":"ContainerDied","Data":"e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb"}
Sep 10 13:53:58.645205 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.645157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8" event={"ID":"d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839","Type":"ContainerDied","Data":"c5cf46571153212d3b9435209faca84744789d8af3fae833fc5c309a237a255a"}
Sep 10 13:53:58.645205 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.645179    2577 scope.go:117] "RemoveContainer" containerID="e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb"
Sep 10 13:53:58.654622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.654601    2577 scope.go:117] "RemoveContainer" containerID="27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"
Sep 10 13:53:58.662797 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.662693    2577 scope.go:117] "RemoveContainer" containerID="e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb"
Sep 10 13:53:58.663191 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:53:58.663028    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb\": container with ID starting with e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb not found: ID does not exist" containerID="e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb"
Sep 10 13:53:58.663191 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.663064    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb"} err="failed to get container status \"e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb\": rpc error: code = NotFound desc = could not find container \"e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb\": container with ID starting with e6805bc8b01be22c2c17053cb3a4736dd2b0c3d7ba4a1e97fa497ef2f9b6e8eb not found: ID does not exist"
Sep 10 13:53:58.663191 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.663085    2577 scope.go:117] "RemoveContainer" containerID="27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"
Sep 10 13:53:58.663596 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:53:58.663551    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036\": container with ID starting with 27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036 not found: ID does not exist" containerID="27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"
Sep 10 13:53:58.663680 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.663610    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036"} err="failed to get container status \"27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036\": rpc error: code = NotFound desc = could not find container \"27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036\": container with ID starting with 27e780a300cc2f963ebfdaa6c789c9454e0cf9a97167ee7124e2d42fd952e036 not found: ID does not exist"
Sep 10 13:53:58.665768 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.665749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"]
Sep 10 13:53:58.669733 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:53:58.669711    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a6883ccf-openshift-default-79d7dd8984-xk9l8"]
Sep 10 13:54:00.452886 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:00.452855    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" path="/var/lib/kubelet/pods/d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839/volumes"
Sep 10 13:54:06.511323 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:06.511286    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"]
Sep 10 13:54:21.531755 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.531695    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng" podUID="ba194fb6-caf7-4519-9a2b-2827245b6649" containerName="main" containerID="cri-o://52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9" gracePeriod=45
Sep 10 13:54:21.788477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.788409    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:54:21.789007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.788983    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba194fb6-caf7-4519-9a2b-2827245b6649" containerID="52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9" exitCode=0
Sep 10 13:54:21.789118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.789044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng" event={"ID":"ba194fb6-caf7-4519-9a2b-2827245b6649","Type":"ContainerDied","Data":"52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9"}
Sep 10 13:54:21.789118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.789077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng" event={"ID":"ba194fb6-caf7-4519-9a2b-2827245b6649","Type":"ContainerDied","Data":"fe93a462d76df783f57ec9c61d8e9e49a6cc1330d2e2e2590f0ffc17f8dd0227"}
Sep 10 13:54:21.789118 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.789093    2577 scope.go:117] "RemoveContainer" containerID="52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9"
Sep 10 13:54:21.797778 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.797754    2577 scope.go:117] "RemoveContainer" containerID="52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9"
Sep 10 13:54:21.798108 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:54:21.798079    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9\": container with ID starting with 52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9 not found: ID does not exist" containerID="52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9"
Sep 10 13:54:21.798196 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.798120    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9"} err="failed to get container status \"52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9\": rpc error: code = NotFound desc = could not find container \"52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9\": container with ID starting with 52d88588b14df5efdd92e20c43a4946a5fd68615b35e71f4638cc244ba6e7ea9 not found: ID does not exist"
Sep 10 13:54:21.904499 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-model-cache\") pod \"ba194fb6-caf7-4519-9a2b-2827245b6649\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") "
Sep 10 13:54:21.904499 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-dshm\") pod \"ba194fb6-caf7-4519-9a2b-2827245b6649\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") "
Sep 10 13:54:21.904793 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-home\") pod \"ba194fb6-caf7-4519-9a2b-2827245b6649\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") "
Sep 10 13:54:21.904793 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904567    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba194fb6-caf7-4519-9a2b-2827245b6649-tls-certs\") pod \"ba194fb6-caf7-4519-9a2b-2827245b6649\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") "
Sep 10 13:54:21.904793 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904664    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-tmp-dir\") pod \"ba194fb6-caf7-4519-9a2b-2827245b6649\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") "
Sep 10 13:54:21.904793 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64zlq\" (UniqueName: \"kubernetes.io/projected/ba194fb6-caf7-4519-9a2b-2827245b6649-kube-api-access-64zlq\") pod \"ba194fb6-caf7-4519-9a2b-2827245b6649\" (UID: \"ba194fb6-caf7-4519-9a2b-2827245b6649\") "
Sep 10 13:54:21.904997 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-home" (OuterVolumeSpecName: "home") pod "ba194fb6-caf7-4519-9a2b-2827245b6649" (UID: "ba194fb6-caf7-4519-9a2b-2827245b6649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:21.904997 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba194fb6-caf7-4519-9a2b-2827245b6649" (UID: "ba194fb6-caf7-4519-9a2b-2827245b6649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:21.904997 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.904912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba194fb6-caf7-4519-9a2b-2827245b6649" (UID: "ba194fb6-caf7-4519-9a2b-2827245b6649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:21.905155 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.905116    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-model-cache\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:54:21.905155 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.905133    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-home\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:54:21.905155 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.905143    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-tmp-dir\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:54:21.906892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.906866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba194fb6-caf7-4519-9a2b-2827245b6649-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba194fb6-caf7-4519-9a2b-2827245b6649" (UID: "ba194fb6-caf7-4519-9a2b-2827245b6649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:54:21.907006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.906913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba194fb6-caf7-4519-9a2b-2827245b6649-kube-api-access-64zlq" (OuterVolumeSpecName: "kube-api-access-64zlq") pod "ba194fb6-caf7-4519-9a2b-2827245b6649" (UID: "ba194fb6-caf7-4519-9a2b-2827245b6649"). InnerVolumeSpecName "kube-api-access-64zlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:54:21.907006 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:21.906943    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-dshm" (OuterVolumeSpecName: "dshm") pod "ba194fb6-caf7-4519-9a2b-2827245b6649" (UID: "ba194fb6-caf7-4519-9a2b-2827245b6649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:22.005757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:22.005722    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zlq\" (UniqueName: \"kubernetes.io/projected/ba194fb6-caf7-4519-9a2b-2827245b6649-kube-api-access-64zlq\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:54:22.005757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:22.005749    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba194fb6-caf7-4519-9a2b-2827245b6649-dshm\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:54:22.005757 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:22.005760    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba194fb6-caf7-4519-9a2b-2827245b6649-tls-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:54:22.796028 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:22.795943    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"
Sep 10 13:54:22.828641 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:22.828599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"]
Sep 10 13:54:22.834159 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:22.834129    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5c012/auth-test-model-d5c012-kserve-6d876dbbbc-zdsng"]
Sep 10 13:54:24.455139 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:24.455092    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba194fb6-caf7-4519-9a2b-2827245b6649" path="/var/lib/kubelet/pods/ba194fb6-caf7-4519-9a2b-2827245b6649/volumes"
Sep 10 13:54:59.518883 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.518848    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"]
Sep 10 13:54:59.520240 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520213    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba194fb6-caf7-4519-9a2b-2827245b6649" containerName="main"
Sep 10 13:54:59.520240 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520237    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba194fb6-caf7-4519-9a2b-2827245b6649" containerName="main"
Sep 10 13:54:59.520435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520263    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy"
Sep 10 13:54:59.520435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520268    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy"
Sep 10 13:54:59.520435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520279    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy"
Sep 10 13:54:59.520435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520284    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy"
Sep 10 13:54:59.520435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520356    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy"
Sep 10 13:54:59.520435 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.520367    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba194fb6-caf7-4519-9a2b-2827245b6649" containerName="main"
Sep 10 13:54:59.523613 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.523571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.526154 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.526131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d5fa55\""
Sep 10 13:54:59.526376 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.526362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d5fa55-dockercfg-7btqq\""
Sep 10 13:54:59.531463 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.531203    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"]
Sep 10 13:54:59.531820 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.531799    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c9f061-9ed9-4b4e-acab-bf1a4a5ef839" containerName="istio-proxy"
Sep 10 13:54:59.534749 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.534729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.539079 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.539049    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"]
Sep 10 13:54:59.552491 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.552469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"]
Sep 10 13:54:59.593797 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.593766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464bcb11-76a1-47bc-88eb-86ff89ccc635-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.593941 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.593831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224ec18d-d9f3-476c-90c8-060ff3726918-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.593941 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.593850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmnj\" (UniqueName: \"kubernetes.io/projected/224ec18d-d9f3-476c-90c8-060ff3726918-kube-api-access-xfmnj\") pod \"payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.593941 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.593913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsl7\" (UniqueName: \"kubernetes.io/projected/464bcb11-76a1-47bc-88eb-86ff89ccc635-kube-api-access-ctsl7\") pod \"payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.695019 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.694981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464bcb11-76a1-47bc-88eb-86ff89ccc635-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.695207 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.695083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224ec18d-d9f3-476c-90c8-060ff3726918-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.695207 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.695105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmnj\" (UniqueName: \"kubernetes.io/projected/224ec18d-d9f3-476c-90c8-060ff3726918-kube-api-access-xfmnj\") pod \"payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.695207 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.695196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsl7\" (UniqueName: \"kubernetes.io/projected/464bcb11-76a1-47bc-88eb-86ff89ccc635-kube-api-access-ctsl7\") pod \"payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.695845 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.695823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464bcb11-76a1-47bc-88eb-86ff89ccc635-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.696016 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.695994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224ec18d-d9f3-476c-90c8-060ff3726918-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.703975 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.703941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsl7\" (UniqueName: \"kubernetes.io/projected/464bcb11-76a1-47bc-88eb-86ff89ccc635-kube-api-access-ctsl7\") pod \"payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:54:59.704166 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.704148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmnj\" (UniqueName: \"kubernetes.io/projected/224ec18d-d9f3-476c-90c8-060ff3726918-kube-api-access-xfmnj\") pod \"payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.835764 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.835729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:54:59.846862 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:54:59.846834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:55:00.130691 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:00.130658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"]
Sep 10 13:55:00.133499 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:55:00.133470    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224ec18d_d9f3_476c_90c8_060ff3726918.slice/crio-43a5b1edeed4ea365747cedc59f8947bf47f24c464b59c9d4356479d9370d937 WatchSource:0}: Error finding container 43a5b1edeed4ea365747cedc59f8947bf47f24c464b59c9d4356479d9370d937: Status 404 returned error can't find the container with id 43a5b1edeed4ea365747cedc59f8947bf47f24c464b59c9d4356479d9370d937
Sep 10 13:55:00.137824 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:00.137804    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"]
Sep 10 13:55:00.138533 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:55:00.138507    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464bcb11_76a1_47bc_88eb_86ff89ccc635.slice/crio-03e0e1268ff938a09fe4e5e1615da88d9ccd3f350549b48766584137f2a086bf WatchSource:0}: Error finding container 03e0e1268ff938a09fe4e5e1615da88d9ccd3f350549b48766584137f2a086bf: Status 404 returned error can't find the container with id 03e0e1268ff938a09fe4e5e1615da88d9ccd3f350549b48766584137f2a086bf
Sep 10 13:55:01.007998 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.007956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d" event={"ID":"224ec18d-d9f3-476c-90c8-060ff3726918","Type":"ContainerStarted","Data":"01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d"}
Sep 10 13:55:01.008459 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.008139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:55:01.008459 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.008181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d" event={"ID":"224ec18d-d9f3-476c-90c8-060ff3726918","Type":"ContainerStarted","Data":"43a5b1edeed4ea365747cedc59f8947bf47f24c464b59c9d4356479d9370d937"}
Sep 10 13:55:01.009551 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.009529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x" event={"ID":"464bcb11-76a1-47bc-88eb-86ff89ccc635","Type":"ContainerStarted","Data":"5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324"}
Sep 10 13:55:01.009695 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.009563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x" event={"ID":"464bcb11-76a1-47bc-88eb-86ff89ccc635","Type":"ContainerStarted","Data":"03e0e1268ff938a09fe4e5e1615da88d9ccd3f350549b48766584137f2a086bf"}
Sep 10 13:55:01.009737 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.009694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:55:01.026825 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.026771    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d" podStartSLOduration=2.026751121 podStartE2EDuration="2.026751121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:01.025768542 +0000 UTC m=+1445.050583243" watchObservedRunningTime="2026-09-10 13:55:01.026751121 +0000 UTC m=+1445.051565829"
Sep 10 13:55:01.046182 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:01.046128    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x" podStartSLOduration=2.046114198 podStartE2EDuration="2.046114198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:01.044666019 +0000 UTC m=+1445.069480725" watchObservedRunningTime="2026-09-10 13:55:01.046114198 +0000 UTC m=+1445.070928901"
Sep 10 13:55:12.017783 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:12.017749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:55:12.018193 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:12.017824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:55:24.968094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:24.968057    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"]
Sep 10 13:55:24.972098 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:24.972078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:24.977164 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:24.977138    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fc1974-openshift-default-dockercfg-vtcwl\""
Sep 10 13:55:25.059243 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.059202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"]
Sep 10 13:55:25.167374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2kh\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-kube-api-access-2q2kh\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167374 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-credential-socket\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-podinfo\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-envoy\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-certs\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167662 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-data\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167853 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e1dc6-f2d9-4eda-9489-367ace02add9-istiod-ca-cert\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167853 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-socket\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.167853 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.167739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-token\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.268675 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-token\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.268675 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2kh\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-kube-api-access-2q2kh\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.268931 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-credential-socket\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.268931 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-podinfo\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.268931 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-envoy\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.268931 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-certs\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.268987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-data\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e1dc6-f2d9-4eda-9489-367ace02add9-istiod-ca-cert\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-socket\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269211 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-credential-socket\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269372 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-certs\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269414 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-data\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269487 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-socket\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.269801 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.269774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e1dc6-f2d9-4eda-9489-367ace02add9-istiod-ca-cert\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.271294 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.271271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-envoy\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.271552 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.271534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-podinfo\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.279377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.279350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-token\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.279483 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.279423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2kh\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-kube-api-access-2q2kh\") pod \"e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") " pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.283906 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.283881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:25.579839 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.571716    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"]
Sep 10 13:55:25.585497 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.585459    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:55:25.585617 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.585532    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:55:25.585617 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:25.585561    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:55:26.157317 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:26.157276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" event={"ID":"e30e1dc6-f2d9-4eda-9489-367ace02add9","Type":"ContainerStarted","Data":"e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"}
Sep 10 13:55:26.157741 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:26.157335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" event={"ID":"e30e1dc6-f2d9-4eda-9489-367ace02add9","Type":"ContainerStarted","Data":"486f907d2c0fd044be5e3d698cd6a80876603202936f7c18384122500a09e27a"}
Sep 10 13:55:26.284650 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:26.284615    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:26.289795 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:26.289766    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:26.323980 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:26.323910    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podStartSLOduration=2.323890633 podStartE2EDuration="2.323890633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:26.187725871 +0000 UTC m=+1470.212540574" watchObservedRunningTime="2026-09-10 13:55:26.323890633 +0000 UTC m=+1470.348705335"
Sep 10 13:55:27.163203 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:27.163170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:27.164187 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:27.164168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:35.620873 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.620829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"]
Sep 10 13:55:35.625401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.625348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.628377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.628347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fc1974-dockercfg-q75qz\""
Sep 10 13:55:35.628377 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.628380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fc1974\""
Sep 10 13:55:35.638677 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.635803    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"]
Sep 10 13:55:35.681521 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.681481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ptm\" (UniqueName: \"kubernetes.io/projected/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-kube-api-access-27ptm\") pod \"payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.681823 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.681790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.782743 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.782698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.782928 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.782767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ptm\" (UniqueName: \"kubernetes.io/projected/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-kube-api-access-27ptm\") pod \"payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.783507 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.783482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.798017 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.797975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ptm\" (UniqueName: \"kubernetes.io/projected/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-kube-api-access-27ptm\") pod \"payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:35.943441 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:35.943343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:36.237989 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:36.237952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"]
Sep 10 13:55:36.238109 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:55:36.238021    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6cddc7_cecd_4fae_8b8b_ff9200da94f2.slice/crio-e769cacf72e223aa0a967ba2a5cd57c11d5915c5055d0ebd2173d61f085a52cd WatchSource:0}: Error finding container e769cacf72e223aa0a967ba2a5cd57c11d5915c5055d0ebd2173d61f085a52cd: Status 404 returned error can't find the container with id e769cacf72e223aa0a967ba2a5cd57c11d5915c5055d0ebd2173d61f085a52cd
Sep 10 13:55:37.227234 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:37.227191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75" event={"ID":"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2","Type":"ContainerStarted","Data":"139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473"}
Sep 10 13:55:37.227749 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:37.227303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:37.227749 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:37.227322    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75" event={"ID":"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2","Type":"ContainerStarted","Data":"e769cacf72e223aa0a967ba2a5cd57c11d5915c5055d0ebd2173d61f085a52cd"}
Sep 10 13:55:37.248337 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:37.248272    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75" podStartSLOduration=2.248254914 podStartE2EDuration="2.248254914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:37.246105899 +0000 UTC m=+1481.270920602" watchObservedRunningTime="2026-09-10 13:55:37.248254914 +0000 UTC m=+1481.273069616"
Sep 10 13:55:48.236165 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:48.236121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:55:54.330712 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:54.330674    2577 generic.go:356] "Generic (PLEG): container finished" podID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerID="e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0" exitCode=0
Sep 10 13:55:54.330712 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:54.330716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" event={"ID":"e30e1dc6-f2d9-4eda-9489-367ace02add9","Type":"ContainerDied","Data":"e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"}
Sep 10 13:55:54.331202 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:54.331135    2577 scope.go:117] "RemoveContainer" containerID="e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"
Sep 10 13:55:54.332334 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:54.332298    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:55:54.332422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:54.332370    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:55:54.332422 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:54.332402    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:55:55.338103 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:55.338063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" event={"ID":"e30e1dc6-f2d9-4eda-9489-367ace02add9","Type":"ContainerStarted","Data":"63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf"}
Sep 10 13:55:56.285135 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:56.285087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:56.286558 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:56.286529    2577 patch_prober.go:28] interesting pod/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 13:55:56.286721 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:56.286612    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 13:55:56.565496 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:56.565408    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:55:56.566841 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:56.566812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 13:55:57.164116 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:57.164070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:55:57.284302 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:57.284258    2577 patch_prober.go:28] interesting pod/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 13:55:57.284482 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:57.284339    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 13:55:58.285248 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:58.285206    2577 patch_prober.go:28] interesting pod/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 13:55:58.285808 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:58.285275    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 13:55:59.285036 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:59.285000    2577 patch_prober.go:28] interesting pod/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 13:55:59.285231 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:55:59.285065    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 13:56:00.284932 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:00.284887    2577 patch_prober.go:28] interesting pod/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 10 13:56:00.285389 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:00.284979    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 10 13:56:01.381493 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:01.381433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:56:01.383307 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:01.383280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:56:17.548599 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.548546    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"]
Sep 10 13:56:17.550353 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.550295    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x" podUID="464bcb11-76a1-47bc-88eb-86ff89ccc635" containerName="payload-processing" containerID="cri-o://5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324" gracePeriod=30
Sep 10 13:56:17.558287 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.558247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"]
Sep 10 13:56:17.558818 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.558789    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d" podUID="224ec18d-d9f3-476c-90c8-060ff3726918" containerName="payload-pre-processing" containerID="cri-o://01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d" gracePeriod=30
Sep 10 13:56:17.841239 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.841203    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:56:17.860779 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.860725    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:56:17.987881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.987839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfmnj\" (UniqueName: \"kubernetes.io/projected/224ec18d-d9f3-476c-90c8-060ff3726918-kube-api-access-xfmnj\") pod \"224ec18d-d9f3-476c-90c8-060ff3726918\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") "
Sep 10 13:56:17.987881 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.987889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224ec18d-d9f3-476c-90c8-060ff3726918-plugins-config-volume\") pod \"224ec18d-d9f3-476c-90c8-060ff3726918\" (UID: \"224ec18d-d9f3-476c-90c8-060ff3726918\") "
Sep 10 13:56:17.988145 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.987959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctsl7\" (UniqueName: \"kubernetes.io/projected/464bcb11-76a1-47bc-88eb-86ff89ccc635-kube-api-access-ctsl7\") pod \"464bcb11-76a1-47bc-88eb-86ff89ccc635\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") "
Sep 10 13:56:17.988145 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.987999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464bcb11-76a1-47bc-88eb-86ff89ccc635-plugins-config-volume\") pod \"464bcb11-76a1-47bc-88eb-86ff89ccc635\" (UID: \"464bcb11-76a1-47bc-88eb-86ff89ccc635\") "
Sep 10 13:56:17.988341 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.988284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224ec18d-d9f3-476c-90c8-060ff3726918-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "224ec18d-d9f3-476c-90c8-060ff3726918" (UID: "224ec18d-d9f3-476c-90c8-060ff3726918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:56:17.988463 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.988441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464bcb11-76a1-47bc-88eb-86ff89ccc635-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "464bcb11-76a1-47bc-88eb-86ff89ccc635" (UID: "464bcb11-76a1-47bc-88eb-86ff89ccc635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:56:17.990083 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.990056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464bcb11-76a1-47bc-88eb-86ff89ccc635-kube-api-access-ctsl7" (OuterVolumeSpecName: "kube-api-access-ctsl7") pod "464bcb11-76a1-47bc-88eb-86ff89ccc635" (UID: "464bcb11-76a1-47bc-88eb-86ff89ccc635"). InnerVolumeSpecName "kube-api-access-ctsl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:17.990168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:17.990109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224ec18d-d9f3-476c-90c8-060ff3726918-kube-api-access-xfmnj" (OuterVolumeSpecName: "kube-api-access-xfmnj") pod "224ec18d-d9f3-476c-90c8-060ff3726918" (UID: "224ec18d-d9f3-476c-90c8-060ff3726918"). InnerVolumeSpecName "kube-api-access-xfmnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:18.088984 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.088949    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmnj\" (UniqueName: \"kubernetes.io/projected/224ec18d-d9f3-476c-90c8-060ff3726918-kube-api-access-xfmnj\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.088984 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.088982    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224ec18d-d9f3-476c-90c8-060ff3726918-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.088984 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.088992    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctsl7\" (UniqueName: \"kubernetes.io/projected/464bcb11-76a1-47bc-88eb-86ff89ccc635-kube-api-access-ctsl7\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.089225 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.089001    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464bcb11-76a1-47bc-88eb-86ff89ccc635-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.484072 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.484029    2577 generic.go:356] "Generic (PLEG): container finished" podID="224ec18d-d9f3-476c-90c8-060ff3726918" containerID="01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d" exitCode=0
Sep 10 13:56:18.484269 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.484093    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"
Sep 10 13:56:18.484269 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.484116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d" event={"ID":"224ec18d-d9f3-476c-90c8-060ff3726918","Type":"ContainerDied","Data":"01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d"}
Sep 10 13:56:18.484269 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.484180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d" event={"ID":"224ec18d-d9f3-476c-90c8-060ff3726918","Type":"ContainerDied","Data":"43a5b1edeed4ea365747cedc59f8947bf47f24c464b59c9d4356479d9370d937"}
Sep 10 13:56:18.484269 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.484209    2577 scope.go:117] "RemoveContainer" containerID="01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d"
Sep 10 13:56:18.485622 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.485567    2577 generic.go:356] "Generic (PLEG): container finished" podID="464bcb11-76a1-47bc-88eb-86ff89ccc635" containerID="5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324" exitCode=0
Sep 10 13:56:18.485722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.485624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x" event={"ID":"464bcb11-76a1-47bc-88eb-86ff89ccc635","Type":"ContainerDied","Data":"5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324"}
Sep 10 13:56:18.485722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.485647    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"
Sep 10 13:56:18.485722 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.485692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x" event={"ID":"464bcb11-76a1-47bc-88eb-86ff89ccc635","Type":"ContainerDied","Data":"03e0e1268ff938a09fe4e5e1615da88d9ccd3f350549b48766584137f2a086bf"}
Sep 10 13:56:18.494652 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.494625    2577 scope.go:117] "RemoveContainer" containerID="01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d"
Sep 10 13:56:18.495383 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:56:18.495328    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d\": container with ID starting with 01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d not found: ID does not exist" containerID="01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d"
Sep 10 13:56:18.495383 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.495360    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d"} err="failed to get container status \"01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d\": rpc error: code = NotFound desc = could not find container \"01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d\": container with ID starting with 01ed16cb77c1022b49bdac1e0c85eabfb2b620ec4f17ae4f2017f64a5e4c218d not found: ID does not exist"
Sep 10 13:56:18.495383 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.495383    2577 scope.go:117] "RemoveContainer" containerID="5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324"
Sep 10 13:56:18.504323 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.504299    2577 scope.go:117] "RemoveContainer" containerID="5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324"
Sep 10 13:56:18.504731 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:56:18.504712    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324\": container with ID starting with 5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324 not found: ID does not exist" containerID="5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324"
Sep 10 13:56:18.504796 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.504739    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324"} err="failed to get container status \"5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324\": rpc error: code = NotFound desc = could not find container \"5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324\": container with ID starting with 5880cfc1f4dd7c5889ed932065800005d7cb815a2f6e957b0b9580597261b324 not found: ID does not exist"
Sep 10 13:56:18.512614 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.512560    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"]
Sep 10 13:56:18.515889 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.515861    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5fa55-94d5976bb-dvx9d"]
Sep 10 13:56:18.533130 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.533095    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"]
Sep 10 13:56:18.538277 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:18.538247    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5fa55-5c75fbf9d7-bcw5x"]
Sep 10 13:56:20.453428 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:20.453326    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224ec18d-d9f3-476c-90c8-060ff3726918" path="/var/lib/kubelet/pods/224ec18d-d9f3-476c-90c8-060ff3726918/volumes"
Sep 10 13:56:20.453946 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:20.453925    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464bcb11-76a1-47bc-88eb-86ff89ccc635" path="/var/lib/kubelet/pods/464bcb11-76a1-47bc-88eb-86ff89ccc635/volumes"
Sep 10 13:56:42.727892 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:42.727845    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"]
Sep 10 13:56:42.728477 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:42.728173    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75" podUID="ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" containerName="payload-processing" containerID="cri-o://139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473" gracePeriod=30
Sep 10 13:56:43.002213 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.002185    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:56:43.061027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.060124    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-plugins-config-volume\") pod \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") "
Sep 10 13:56:43.061027 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.060214    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27ptm\" (UniqueName: \"kubernetes.io/projected/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-kube-api-access-27ptm\") pod \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\" (UID: \"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2\") "
Sep 10 13:56:43.061910 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.061862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" (UID: "ec6cddc7-cecd-4fae-8b8b-ff9200da94f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:56:43.080137 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.080095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-kube-api-access-27ptm" (OuterVolumeSpecName: "kube-api-access-27ptm") pod "ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" (UID: "ec6cddc7-cecd-4fae-8b8b-ff9200da94f2"). InnerVolumeSpecName "kube-api-access-27ptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:43.161243 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.161202    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:43.161243 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.161238    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27ptm\" (UniqueName: \"kubernetes.io/projected/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2-kube-api-access-27ptm\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:43.639388 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.639346    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" containerID="139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473" exitCode=0
Sep 10 13:56:43.639388 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.639393    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"
Sep 10 13:56:43.639651 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.639433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75" event={"ID":"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2","Type":"ContainerDied","Data":"139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473"}
Sep 10 13:56:43.639651 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.639491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75" event={"ID":"ec6cddc7-cecd-4fae-8b8b-ff9200da94f2","Type":"ContainerDied","Data":"e769cacf72e223aa0a967ba2a5cd57c11d5915c5055d0ebd2173d61f085a52cd"}
Sep 10 13:56:43.639651 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.639513    2577 scope.go:117] "RemoveContainer" containerID="139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473"
Sep 10 13:56:43.649379 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.649348    2577 scope.go:117] "RemoveContainer" containerID="139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473"
Sep 10 13:56:43.649929 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:56:43.649886    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473\": container with ID starting with 139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473 not found: ID does not exist" containerID="139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473"
Sep 10 13:56:43.649929 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.649920    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473"} err="failed to get container status \"139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473\": rpc error: code = NotFound desc = could not find container \"139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473\": container with ID starting with 139d18e5f7e63957cb4b1cca1170c816895fdcdfd824047dfd127b8bbf4f9473 not found: ID does not exist"
Sep 10 13:56:43.663313 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.663276    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"]
Sep 10 13:56:43.668163 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:43.668131    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fc1974-5d57bb757-x9n75"]
Sep 10 13:56:44.453521 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:44.453474    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" path="/var/lib/kubelet/pods/ec6cddc7-cecd-4fae-8b8b-ff9200da94f2/volumes"
Sep 10 13:56:50.412288 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:50.412241    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"]
Sep 10 13:56:50.412826 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:50.412539    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy" containerID="cri-o://63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf" gracePeriod=30
Sep 10 13:56:55.678270 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.678192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:56:55.717492 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.717450    2577 generic.go:356] "Generic (PLEG): container finished" podID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerID="63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf" exitCode=0
Sep 10 13:56:55.717694 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.717507    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"
Sep 10 13:56:55.717694 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.717541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" event={"ID":"e30e1dc6-f2d9-4eda-9489-367ace02add9","Type":"ContainerDied","Data":"63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf"}
Sep 10 13:56:55.717694 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.717634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q" event={"ID":"e30e1dc6-f2d9-4eda-9489-367ace02add9","Type":"ContainerDied","Data":"486f907d2c0fd044be5e3d698cd6a80876603202936f7c18384122500a09e27a"}
Sep 10 13:56:55.717694 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.717652    2577 scope.go:117] "RemoveContainer" containerID="63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf"
Sep 10 13:56:55.729672 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.729639    2577 scope.go:117] "RemoveContainer" containerID="e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"
Sep 10 13:56:55.740864 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.740838    2577 scope.go:117] "RemoveContainer" containerID="63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf"
Sep 10 13:56:55.741224 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:56:55.741201    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf\": container with ID starting with 63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf not found: ID does not exist" containerID="63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf"
Sep 10 13:56:55.741327 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.741238    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf"} err="failed to get container status \"63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf\": rpc error: code = NotFound desc = could not find container \"63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf\": container with ID starting with 63bdd920fb880a5915d10970bc5dfe337f33c793a97f5a5bddb87bbe81bf8aaf not found: ID does not exist"
Sep 10 13:56:55.741327 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.741264    2577 scope.go:117] "RemoveContainer" containerID="e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"
Sep 10 13:56:55.741647 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:56:55.741619    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0\": container with ID starting with e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0 not found: ID does not exist" containerID="e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"
Sep 10 13:56:55.741703 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.741659    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0"} err="failed to get container status \"e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0\": rpc error: code = NotFound desc = could not find container \"e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0\": container with ID starting with e08343755870d5d6e39ae2232d86c2cbdfc90c411f747c88f34aa3222cbe7bb0 not found: ID does not exist"
Sep 10 13:56:55.803936 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.803898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-data\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.803936 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.803948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q2kh\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-kube-api-access-2q2kh\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804198 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804012    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-certs\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804198 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-socket\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804198 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-credential-socket\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804198 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-envoy\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804198 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e1dc6-f2d9-4eda-9489-367ace02add9-istiod-ca-cert\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804198 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-token\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804481 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804264    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-data" (OuterVolumeSpecName: "istio-data") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:55.804481 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:55.804481 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804370    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-podinfo\") pod \"e30e1dc6-f2d9-4eda-9489-367ace02add9\" (UID: \"e30e1dc6-f2d9-4eda-9489-367ace02add9\") "
Sep 10 13:56:55.804630 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:55.804630 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:55.804848 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804825    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.804913 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804861    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.804913 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804875    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.804913 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804891    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.804913 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.804815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30e1dc6-f2d9-4eda-9489-367ace02add9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:56:55.806483 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.806452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:55.806691 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.806552    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:56:55.806691 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.806617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-kube-api-access-2q2kh" (OuterVolumeSpecName: "kube-api-access-2q2kh") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "kube-api-access-2q2kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:55.806894 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.806864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-token" (OuterVolumeSpecName: "istio-token") pod "e30e1dc6-f2d9-4eda-9489-367ace02add9" (UID: "e30e1dc6-f2d9-4eda-9489-367ace02add9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:55.905947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.905899    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2kh\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-kube-api-access-2q2kh\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.905947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.905940    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.905947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.905950    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e1dc6-f2d9-4eda-9489-367ace02add9-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.905947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.905958    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:55.906233 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:55.905967    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e1dc6-f2d9-4eda-9489-367ace02add9-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:56:56.046807 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:56.046763    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"]
Sep 10 13:56:56.050600 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:56.050549    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fc1974-openshift-default-59b6546c94-kdq8q"]
Sep 10 13:56:56.453309 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:56:56.453272    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" path="/var/lib/kubelet/pods/e30e1dc6-f2d9-4eda-9489-367ace02add9/volumes"
Sep 10 13:57:01.436500 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.436457    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"]
Sep 10 13:57:01.436948 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.436695    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2" podUID="9559de0b-ddd6-4ab2-8dce-ee31e32114bc" containerName="maas-api" containerID="cri-o://ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0" gracePeriod=30
Sep 10 13:57:01.704538 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.704507    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:57:01.755244 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.755208    2577 generic.go:356] "Generic (PLEG): container finished" podID="9559de0b-ddd6-4ab2-8dce-ee31e32114bc" containerID="ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0" exitCode=0
Sep 10 13:57:01.755450 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.755263    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"
Sep 10 13:57:01.755450 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.755293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2" event={"ID":"9559de0b-ddd6-4ab2-8dce-ee31e32114bc","Type":"ContainerDied","Data":"ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0"}
Sep 10 13:57:01.755450 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.755377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2" event={"ID":"9559de0b-ddd6-4ab2-8dce-ee31e32114bc","Type":"ContainerDied","Data":"ff8555068ec3f0ef3678d3f108a5458bed874757290c2855f21fbb3cf783c4bc"}
Sep 10 13:57:01.755450 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.755401    2577 scope.go:117] "RemoveContainer" containerID="ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0"
Sep 10 13:57:01.765452 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.765430    2577 scope.go:117] "RemoveContainer" containerID="ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0"
Sep 10 13:57:01.765845 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:57:01.765816    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0\": container with ID starting with ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0 not found: ID does not exist" containerID="ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0"
Sep 10 13:57:01.765934 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.765850    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0"} err="failed to get container status \"ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0\": rpc error: code = NotFound desc = could not find container \"ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0\": container with ID starting with ff0d2197a9ffebf7e67aa133a39a1b8e34539516681ead1462a2cc3ea9ffb7b0 not found: ID does not exist"
Sep 10 13:57:01.868155 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.868111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-metrics-certs\") pod \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") "
Sep 10 13:57:01.868390 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.868175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-openshift-service-ca\") pod \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") "
Sep 10 13:57:01.868390 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.868293    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7r58\" (UniqueName: \"kubernetes.io/projected/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-kube-api-access-t7r58\") pod \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") "
Sep 10 13:57:01.868517 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.868410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-maas-api-tls\") pod \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\" (UID: \"9559de0b-ddd6-4ab2-8dce-ee31e32114bc\") "
Sep 10 13:57:01.869333 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.869295    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9559de0b-ddd6-4ab2-8dce-ee31e32114bc" (UID: "9559de0b-ddd6-4ab2-8dce-ee31e32114bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:57:01.872062 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.872005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-kube-api-access-t7r58" (OuterVolumeSpecName: "kube-api-access-t7r58") pod "9559de0b-ddd6-4ab2-8dce-ee31e32114bc" (UID: "9559de0b-ddd6-4ab2-8dce-ee31e32114bc"). InnerVolumeSpecName "kube-api-access-t7r58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:57:01.872223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.872138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9559de0b-ddd6-4ab2-8dce-ee31e32114bc" (UID: "9559de0b-ddd6-4ab2-8dce-ee31e32114bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:57:01.872456 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.872433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9559de0b-ddd6-4ab2-8dce-ee31e32114bc" (UID: "9559de0b-ddd6-4ab2-8dce-ee31e32114bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:57:01.969961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.969863    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7r58\" (UniqueName: \"kubernetes.io/projected/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-kube-api-access-t7r58\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:01.969961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.969903    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:01.969961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.969912    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:01.969961 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:01.969920    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9559de0b-ddd6-4ab2-8dce-ee31e32114bc-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:02.094918 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:02.094876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"]
Sep 10 13:57:02.106052 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:02.106016    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d0caa-769d796c46-4z6z2"]
Sep 10 13:57:02.453169 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:02.453130    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9559de0b-ddd6-4ab2-8dce-ee31e32114bc" path="/var/lib/kubelet/pods/9559de0b-ddd6-4ab2-8dce-ee31e32114bc/volumes"
Sep 10 13:57:10.090010 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:10.089963    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"]
Sep 10 13:57:10.090499 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:10.090295    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" containerID="cri-o://501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3" gracePeriod=30
Sep 10 13:57:11.754656 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:11.754619    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:57:11.755050 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:11.754697    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:57:15.360970 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.360889    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:57:15.410417 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-socket\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410417 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-certs\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410661 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-podinfo\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410661 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-envoy\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410747 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-token\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410747 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-credential-socket\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410843 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:57:15.410843 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-data\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410843 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410831    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:57:15.410994 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kshg9\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-kube-api-access-kshg9\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.410994 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.410934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istiod-ca-cert\") pod \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\" (UID: \"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b\") "
Sep 10 13:57:15.411082 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:57:15.411082 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:57:15.411389 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411348    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.411389 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411383    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:57:15.411609 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411391    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.411609 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411433    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.411609 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.411452    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.412965 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.412918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:57:15.413073 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.412938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-kube-api-access-kshg9" (OuterVolumeSpecName: "kube-api-access-kshg9") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "kube-api-access-kshg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:57:15.413073 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.412967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:57:15.413073 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.413004    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" (UID: "6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:57:15.512855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.512809    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.512855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.512845    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshg9\" (UniqueName: \"kubernetes.io/projected/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-kube-api-access-kshg9\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.512855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.512855    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.512855 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.512864    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.513140 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.512872    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:15.851497 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.851457    2577 generic.go:356] "Generic (PLEG): container finished" podID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerID="501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3" exitCode=0
Sep 10 13:57:15.851497 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.851502    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"
Sep 10 13:57:15.851758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.851539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" event={"ID":"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b","Type":"ContainerDied","Data":"501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3"}
Sep 10 13:57:15.851758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.851631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj" event={"ID":"6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b","Type":"ContainerDied","Data":"864064a7527132102914b3df278756c9ba34883e445983e7ea417681562dc61a"}
Sep 10 13:57:15.851758 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.851659    2577 scope.go:117] "RemoveContainer" containerID="501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3"
Sep 10 13:57:15.861716 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.861691    2577 scope.go:117] "RemoveContainer" containerID="2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"
Sep 10 13:57:15.870697 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.870673    2577 scope.go:117] "RemoveContainer" containerID="501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3"
Sep 10 13:57:15.871046 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:57:15.871025    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3\": container with ID starting with 501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3 not found: ID does not exist" containerID="501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3"
Sep 10 13:57:15.871125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.871059    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3"} err="failed to get container status \"501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3\": rpc error: code = NotFound desc = could not find container \"501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3\": container with ID starting with 501340c3cf40f995a20f30320dbff2a55866ee7ac7e1e0d1fda005574970e9c3 not found: ID does not exist"
Sep 10 13:57:15.871125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.871078    2577 scope.go:117] "RemoveContainer" containerID="2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"
Sep 10 13:57:15.871366 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:57:15.871344    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57\": container with ID starting with 2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57 not found: ID does not exist" containerID="2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"
Sep 10 13:57:15.871414 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.871372    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57"} err="failed to get container status \"2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57\": rpc error: code = NotFound desc = could not find container \"2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57\": container with ID starting with 2fd6f14784788ad54898b0490c8e89ec7aeacb0502eb66eb41ce9df3c025be57 not found: ID does not exist"
Sep 10 13:57:15.876301 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.876270    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"]
Sep 10 13:57:15.884264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:15.884233    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d0caa-openshift-default-7f848b5666-f5vgj"]
Sep 10 13:57:16.454279 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:16.454245    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" path="/var/lib/kubelet/pods/6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b/volumes"
Sep 10 13:57:21.442809 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.442767    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"]
Sep 10 13:57:21.443371 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.443091    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" podUID="fb82f7d2-acf3-4173-820f-524de10dec90" containerName="maas-api" containerID="cri-o://ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05" gracePeriod=30
Sep 10 13:57:21.728264 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.728209    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:57:21.777317 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.777283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f7d2-acf3-4173-820f-524de10dec90-openshift-service-ca\") pod \"fb82f7d2-acf3-4173-820f-524de10dec90\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") "
Sep 10 13:57:21.777522 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.777329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpqnp\" (UniqueName: \"kubernetes.io/projected/fb82f7d2-acf3-4173-820f-524de10dec90-kube-api-access-dpqnp\") pod \"fb82f7d2-acf3-4173-820f-524de10dec90\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") "
Sep 10 13:57:21.777522 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.777371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-metrics-certs\") pod \"fb82f7d2-acf3-4173-820f-524de10dec90\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") "
Sep 10 13:57:21.777678 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.777534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls\") pod \"fb82f7d2-acf3-4173-820f-524de10dec90\" (UID: \"fb82f7d2-acf3-4173-820f-524de10dec90\") "
Sep 10 13:57:21.777761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.777731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82f7d2-acf3-4173-820f-524de10dec90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb82f7d2-acf3-4173-820f-524de10dec90" (UID: "fb82f7d2-acf3-4173-820f-524de10dec90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:57:21.778025 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.777991    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f7d2-acf3-4173-820f-524de10dec90-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:21.779800 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.779772    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb82f7d2-acf3-4173-820f-524de10dec90" (UID: "fb82f7d2-acf3-4173-820f-524de10dec90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:57:21.780049 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.780027    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb82f7d2-acf3-4173-820f-524de10dec90-kube-api-access-dpqnp" (OuterVolumeSpecName: "kube-api-access-dpqnp") pod "fb82f7d2-acf3-4173-820f-524de10dec90" (UID: "fb82f7d2-acf3-4173-820f-524de10dec90"). InnerVolumeSpecName "kube-api-access-dpqnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:57:21.780117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.780031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb82f7d2-acf3-4173-820f-524de10dec90" (UID: "fb82f7d2-acf3-4173-820f-524de10dec90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:57:21.879166 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.879123    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpqnp\" (UniqueName: \"kubernetes.io/projected/fb82f7d2-acf3-4173-820f-524de10dec90-kube-api-access-dpqnp\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:21.879166 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.879162    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:21.879166 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.879171    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb82f7d2-acf3-4173-820f-524de10dec90-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 13:57:21.886653 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.886601    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb82f7d2-acf3-4173-820f-524de10dec90" containerID="ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05" exitCode=0
Sep 10 13:57:21.886851 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.886665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"
Sep 10 13:57:21.886851 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.886664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" event={"ID":"fb82f7d2-acf3-4173-820f-524de10dec90","Type":"ContainerDied","Data":"ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05"}
Sep 10 13:57:21.886959 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.886904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm" event={"ID":"fb82f7d2-acf3-4173-820f-524de10dec90","Type":"ContainerDied","Data":"d19fb01b806995929ed1717539c8c254e909c6de2c0438024a55515422f68796"}
Sep 10 13:57:21.887010 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.886963    2577 scope.go:117] "RemoveContainer" containerID="ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05"
Sep 10 13:57:21.899502 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.899476    2577 scope.go:117] "RemoveContainer" containerID="ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05"
Sep 10 13:57:21.899934 ip-10-0-130-202 kubenswrapper[2577]: E0910 13:57:21.899911    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05\": container with ID starting with ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05 not found: ID does not exist" containerID="ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05"
Sep 10 13:57:21.900028 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.899944    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05"} err="failed to get container status \"ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05\": rpc error: code = NotFound desc = could not find container \"ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05\": container with ID starting with ae49cd0db0a25f85570d84b2d7bbe260eacee2db0a8a0efd5dc70a9376df4e05 not found: ID does not exist"
Sep 10 13:57:21.944542 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.944445    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"]
Sep 10 13:57:21.953193 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:21.953157    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b3723e-66466bb9ff-hgwjm"]
Sep 10 13:57:22.453516 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:22.453479    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb82f7d2-acf3-4173-820f-524de10dec90" path="/var/lib/kubelet/pods/fb82f7d2-acf3-4173-820f-524de10dec90/volumes"
Sep 10 13:57:58.687495 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.687459    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"]
Sep 10 13:57:58.688013 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.687993    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy"
Sep 10 13:57:58.688065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688017    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy"
Sep 10 13:57:58.688065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688032    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224ec18d-d9f3-476c-90c8-060ff3726918" containerName="payload-pre-processing"
Sep 10 13:57:58.688065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688041    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="224ec18d-d9f3-476c-90c8-060ff3726918" containerName="payload-pre-processing"
Sep 10 13:57:58.688065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688055    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy"
Sep 10 13:57:58.688065 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688064    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688076    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464bcb11-76a1-47bc-88eb-86ff89ccc635" containerName="payload-processing"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688086    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="464bcb11-76a1-47bc-88eb-86ff89ccc635" containerName="payload-processing"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688107    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb82f7d2-acf3-4173-820f-524de10dec90" containerName="maas-api"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688115    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb82f7d2-acf3-4173-820f-524de10dec90" containerName="maas-api"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688129    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9559de0b-ddd6-4ab2-8dce-ee31e32114bc" containerName="maas-api"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688136    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9559de0b-ddd6-4ab2-8dce-ee31e32114bc" containerName="maas-api"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688151    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" containerName="payload-processing"
Sep 10 13:57:58.688228 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" containerName="payload-processing"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688254    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9559de0b-ddd6-4ab2-8dce-ee31e32114bc" containerName="maas-api"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688265    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688271    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="464bcb11-76a1-47bc-88eb-86ff89ccc635" containerName="payload-processing"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688277    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6cddc7-cecd-4fae-8b8b-ff9200da94f2" containerName="payload-processing"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688284    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688290    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb82f7d2-acf3-4173-820f-524de10dec90" containerName="maas-api"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688298    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688304    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="224ec18d-d9f3-476c-90c8-060ff3726918" containerName="payload-pre-processing"
Sep 10 13:57:58.688471 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.688308    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy"
Sep 10 13:57:58.691443 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.691423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.693946 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.693923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0fe383-openshift-default-dockercfg-pww4g\""
Sep 10 13:57:58.707024 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.706994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"]
Sep 10 13:57:58.763272 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-envoy\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763461 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c592721-527b-4db9-8b60-dc79ed27cb79-istiod-ca-cert\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763461 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-token\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763461 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-certs\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763461 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-credential-socket\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763629 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-socket\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763629 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-podinfo\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763629 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7xft\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-kube-api-access-r7xft\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.763629 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.763574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-data\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.864789 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c592721-527b-4db9-8b60-dc79ed27cb79-istiod-ca-cert\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.864789 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-token\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-certs\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-credential-socket\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-socket\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-podinfo\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xft\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-kube-api-access-r7xft\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.864972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-data\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865053 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.865017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-envoy\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.865302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-socket\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.865328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-credential-socket\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865507 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.865434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-certs\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865568 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.865524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-data\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.865780 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.865759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c592721-527b-4db9-8b60-dc79ed27cb79-istiod-ca-cert\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.867327 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.867308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-envoy\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.867386 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.867371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-podinfo\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.881857 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.881827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-token\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:58.882820 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:58.882798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xft\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-kube-api-access-r7xft\") pod \"e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") " pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:59.002077 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:59.001969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:57:59.289673 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:59.289644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"]
Sep 10 13:57:59.290896 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:57:59.290868    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c592721_527b_4db9_8b60_dc79ed27cb79.slice/crio-b5c5ccf7664c0984c0b27146ef4ded555063efd57223cc5d1df7d23c902861c2 WatchSource:0}: Error finding container b5c5ccf7664c0984c0b27146ef4ded555063efd57223cc5d1df7d23c902861c2: Status 404 returned error can't find the container with id b5c5ccf7664c0984c0b27146ef4ded555063efd57223cc5d1df7d23c902861c2
Sep 10 13:57:59.293381 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:59.293345    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:57:59.293470 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:59.293421    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:57:59.293470 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:57:59.293454    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:00.101263 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:00.101225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" event={"ID":"4c592721-527b-4db9-8b60-dc79ed27cb79","Type":"ContainerStarted","Data":"7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"}
Sep 10 13:58:00.101688 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:00.101279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" event={"ID":"4c592721-527b-4db9-8b60-dc79ed27cb79","Type":"ContainerStarted","Data":"b5c5ccf7664c0984c0b27146ef4ded555063efd57223cc5d1df7d23c902861c2"}
Sep 10 13:58:00.154323 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:00.154264    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podStartSLOduration=2.154244306 podStartE2EDuration="2.154244306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:00.154088155 +0000 UTC m=+1624.178902857" watchObservedRunningTime="2026-09-10 13:58:00.154244306 +0000 UTC m=+1624.179059081"
Sep 10 13:58:01.002774 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:01.002734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:01.008081 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:01.008051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:01.107078 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:01.107043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:01.108084 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:01.108060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:05.672833 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.672789    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"]
Sep 10 13:58:05.673289 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.673275    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy"
Sep 10 13:58:05.673289 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.673289    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be83cce-9d30-43f2-b2f6-a91f6f9e6b3b" containerName="istio-proxy"
Sep 10 13:58:05.673364 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.673305    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy"
Sep 10 13:58:05.673364 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.673314    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e30e1dc6-f2d9-4eda-9489-367ace02add9" containerName="istio-proxy"
Sep 10 13:58:05.676321 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.676298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.679257 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.679233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0fe383\""
Sep 10 13:58:05.697499 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.697462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"]
Sep 10 13:58:05.800087 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.800044    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"]
Sep 10 13:58:05.804573 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.804547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:05.808189 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.808158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0fe383-dockercfg-pjskw\""
Sep 10 13:58:05.813770 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.813735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"]
Sep 10 13:58:05.844215 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.844176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6zk\" (UniqueName: \"kubernetes.io/projected/56738747-e951-4863-b8c7-3afd798eed1e-kube-api-access-zw6zk\") pod \"payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.844402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.844232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56738747-e951-4863-b8c7-3afd798eed1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.944970 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.944882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqpn\" (UniqueName: \"kubernetes.io/projected/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-kube-api-access-dxqpn\") pod \"payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:05.945148 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.944992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:05.945148 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.945083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6zk\" (UniqueName: \"kubernetes.io/projected/56738747-e951-4863-b8c7-3afd798eed1e-kube-api-access-zw6zk\") pod \"payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.945148 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.945117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56738747-e951-4863-b8c7-3afd798eed1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.945903 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.945880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56738747-e951-4863-b8c7-3afd798eed1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.955467 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.955436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6zk\" (UniqueName: \"kubernetes.io/projected/56738747-e951-4863-b8c7-3afd798eed1e-kube-api-access-zw6zk\") pod \"payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:05.987068 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:05.987030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:06.046467 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.046417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:06.046683 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.046607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqpn\" (UniqueName: \"kubernetes.io/projected/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-kube-api-access-dxqpn\") pod \"payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:06.047295 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.047270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:06.062685 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.061303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqpn\" (UniqueName: \"kubernetes.io/projected/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-kube-api-access-dxqpn\") pod \"payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:06.120371 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.120319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:06.313670 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.313214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"]
Sep 10 13:58:06.316655 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:58:06.316562    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56738747_e951_4863_b8c7_3afd798eed1e.slice/crio-204af7d84985bbe60260735646624836d163d6d16c5abef3c22580aa2acea8a5 WatchSource:0}: Error finding container 204af7d84985bbe60260735646624836d163d6d16c5abef3c22580aa2acea8a5: Status 404 returned error can't find the container with id 204af7d84985bbe60260735646624836d163d6d16c5abef3c22580aa2acea8a5
Sep 10 13:58:06.422952 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:06.422916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"]
Sep 10 13:58:06.425280 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:58:06.425244    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c7c51b_6a9c_4a24_9b4b_0c5c8ec0bb98.slice/crio-e641cb7b96ff0fbe6b73e75fd1294c7026bcf9ac5e7601529678a19080a0d9ef WatchSource:0}: Error finding container e641cb7b96ff0fbe6b73e75fd1294c7026bcf9ac5e7601529678a19080a0d9ef: Status 404 returned error can't find the container with id e641cb7b96ff0fbe6b73e75fd1294c7026bcf9ac5e7601529678a19080a0d9ef
Sep 10 13:58:07.143596 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.143537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5" event={"ID":"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98","Type":"ContainerStarted","Data":"2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88"}
Sep 10 13:58:07.144095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.143779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:07.144095 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.143818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5" event={"ID":"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98","Type":"ContainerStarted","Data":"e641cb7b96ff0fbe6b73e75fd1294c7026bcf9ac5e7601529678a19080a0d9ef"}
Sep 10 13:58:07.145326 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.145300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2" event={"ID":"56738747-e951-4863-b8c7-3afd798eed1e","Type":"ContainerStarted","Data":"289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8"}
Sep 10 13:58:07.145465 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.145342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2" event={"ID":"56738747-e951-4863-b8c7-3afd798eed1e","Type":"ContainerStarted","Data":"204af7d84985bbe60260735646624836d163d6d16c5abef3c22580aa2acea8a5"}
Sep 10 13:58:07.145465 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.145368    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:07.166488 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.166419    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5" podStartSLOduration=2.166399131 podStartE2EDuration="2.166399131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:07.164458827 +0000 UTC m=+1631.189273532" watchObservedRunningTime="2026-09-10 13:58:07.166399131 +0000 UTC m=+1631.191213833"
Sep 10 13:58:07.183522 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:07.183461    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2" podStartSLOduration=2.183443685 podStartE2EDuration="2.183443685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:07.181807809 +0000 UTC m=+1631.206622513" watchObservedRunningTime="2026-09-10 13:58:07.183443685 +0000 UTC m=+1631.208258387"
Sep 10 13:58:18.153701 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:18.153663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 13:58:18.154125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:18.153830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 13:58:20.224373 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:20.224264    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerID="7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b" exitCode=0
Sep 10 13:58:20.224373 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:20.224336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" event={"ID":"4c592721-527b-4db9-8b60-dc79ed27cb79","Type":"ContainerDied","Data":"7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"}
Sep 10 13:58:20.224843 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:20.224773    2577 scope.go:117] "RemoveContainer" containerID="7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"
Sep 10 13:58:20.225856 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:20.225826    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:20.225947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:20.225895    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:20.225947 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:20.225928    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:21.237319 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:21.237267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" event={"ID":"4c592721-527b-4db9-8b60-dc79ed27cb79","Type":"ContainerStarted","Data":"b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b"}
Sep 10 13:58:22.002412 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:22.002365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:22.002706 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:22.002681    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 13:58:22.002761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:22.002736    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 13:58:23.002628 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:23.002558    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 13:58:23.003007 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:23.002669    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 13:58:24.003462 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:24.003420    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 13:58:24.003877 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:24.003498    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 13:58:25.002928 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:25.002884    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 13:58:25.003117 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:25.002947    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 13:58:26.002350 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:26.002304    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 13:58:26.002761 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:26.002367    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 13:58:27.007440 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:27.007399    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:27.007842 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:27.007724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:27.008410 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:27.008389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 13:58:28.436110 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.436074    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"]
Sep 10 13:58:28.440090 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.440066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.442619 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.442559    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fc6ea5-openshift-default-dockercfg-vjc7j\""
Sep 10 13:58:28.457174 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.457142    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"]
Sep 10 13:58:28.489240 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-certs\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489240 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln47l\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-kube-api-access-ln47l\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489475 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-credential-socket\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489475 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a359b055-9710-487b-a96a-69d6f91682e1-istio-podinfo\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489475 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a359b055-9710-487b-a96a-69d6f91682e1-istiod-ca-cert\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489475 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-socket\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489644 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-envoy\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489644 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-istio-token\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.489644 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.489566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-data\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590401 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-data\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590643 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-certs\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590643 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln47l\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-kube-api-access-ln47l\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590643 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-credential-socket\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590643 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a359b055-9710-487b-a96a-69d6f91682e1-istio-podinfo\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a359b055-9710-487b-a96a-69d6f91682e1-istiod-ca-cert\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-socket\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-envoy\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.590880 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-istio-token\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.591094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.590880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-data\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.591094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.591046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-credential-socket\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.591094 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.591009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-certs\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.591275 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.591172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-socket\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.591476 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.591446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a359b055-9710-487b-a96a-69d6f91682e1-istiod-ca-cert\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.593024 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.593003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-envoy\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.593284 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.593265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a359b055-9710-487b-a96a-69d6f91682e1-istio-podinfo\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.600369 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.600333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-istio-token\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.600523 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.600417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln47l\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-kube-api-access-ln47l\") pod \"e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") " pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:28.752737 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:28.752644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:29.035860 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.035827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"]
Sep 10 13:58:29.037766 ip-10-0-130-202 kubenswrapper[2577]: W0910 13:58:29.037740    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda359b055_9710_487b_a96a_69d6f91682e1.slice/crio-46f8816d77c896b6fd3fce60ce9410f8458c07963b12d598b204d2e863fec386 WatchSource:0}: Error finding container 46f8816d77c896b6fd3fce60ce9410f8458c07963b12d598b204d2e863fec386: Status 404 returned error can't find the container with id 46f8816d77c896b6fd3fce60ce9410f8458c07963b12d598b204d2e863fec386
Sep 10 13:58:29.040125 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.040087    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:29.040223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.040164    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:29.040223 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.040192    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:29.289752 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.289638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" event={"ID":"a359b055-9710-487b-a96a-69d6f91682e1","Type":"ContainerStarted","Data":"afba5489d8b3d6f70f6dfa5ada17384f26d8636bdc9e30f93a66bf7d5eb0e82a"}
Sep 10 13:58:29.289752 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.289707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" event={"ID":"a359b055-9710-487b-a96a-69d6f91682e1","Type":"ContainerStarted","Data":"46f8816d77c896b6fd3fce60ce9410f8458c07963b12d598b204d2e863fec386"}
Sep 10 13:58:29.316889 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.316695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podStartSLOduration=1.316657145 podStartE2EDuration="1.316657145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:29.315742077 +0000 UTC m=+1653.340556783" watchObservedRunningTime="2026-09-10 13:58:29.316657145 +0000 UTC m=+1653.341471852"
Sep 10 13:58:29.753115 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:29.753070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:30.759104 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:30.759066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:31.308566 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:31.308526    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:31.309701 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:31.309674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:50.418684 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:50.418640    2577 generic.go:356] "Generic (PLEG): container finished" podID="a359b055-9710-487b-a96a-69d6f91682e1" containerID="afba5489d8b3d6f70f6dfa5ada17384f26d8636bdc9e30f93a66bf7d5eb0e82a" exitCode=0
Sep 10 13:58:50.419091 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:50.418709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" event={"ID":"a359b055-9710-487b-a96a-69d6f91682e1","Type":"ContainerDied","Data":"afba5489d8b3d6f70f6dfa5ada17384f26d8636bdc9e30f93a66bf7d5eb0e82a"}
Sep 10 13:58:50.419208 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:50.419193    2577 scope.go:117] "RemoveContainer" containerID="afba5489d8b3d6f70f6dfa5ada17384f26d8636bdc9e30f93a66bf7d5eb0e82a"
Sep 10 13:58:50.420340 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:50.420312    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:50.420402 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:50.420383    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:50.420441 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:50.420417    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:58:51.430119 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:51.430071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" event={"ID":"a359b055-9710-487b-a96a-69d6f91682e1","Type":"ContainerStarted","Data":"f5d88855d40505516efd5bba1f8944c6abe2168979a5a874df0e2d94f7614d11"}
Sep 10 13:58:51.753148 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:51.753061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:51.753322 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:51.753282    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 13:58:51.753394 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:51.753340    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 13:58:52.753168 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:52.753123    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 13:58:52.753705 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:52.753195    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 13:58:53.753487 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:53.753444    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 13:58:53.753946 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:53.753507    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 13:58:54.754068 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:54.754026    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 13:58:54.754496 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:54.754088    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 13:58:55.753770 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:55.753725    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 13:58:55.753967 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:55.753790    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 13:58:56.753266 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:56.753219    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 13:58:56.753751 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:56.753313    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 13:58:57.758147 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:57.758114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:57.758626 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:57.758525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 13:58:57.759248 ip-10-0-130-202 kubenswrapper[2577]: I0910 13:58:57.759228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 14:00:00.141758 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.141724    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"]
Sep 10 14:00:00.146663 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.146637    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"]
Sep 10 14:00:00.146793 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.146760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"
Sep 10 14:00:00.150129 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.150114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"
Sep 10 14:00:00.151604 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.151588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\""
Sep 10 14:00:00.161221 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.161184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"]
Sep 10 14:00:00.161371 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.161351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"
Sep 10 14:00:00.164292 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.164272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"]
Sep 10 14:00:00.169050 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.169032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"
Sep 10 14:00:00.453748 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.453687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"]
Sep 10 14:00:00.454536 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:00:00.454513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad87b86c_7044_432a_8501_2e819ed59540.slice/crio-8cd2601f7aac6930360162907820af1a77613e50c879e1d43dd29e1748a89af6 WatchSource:0}: Error finding container 8cd2601f7aac6930360162907820af1a77613e50c879e1d43dd29e1748a89af6: Status 404 returned error can't find the container with id 8cd2601f7aac6930360162907820af1a77613e50c879e1d43dd29e1748a89af6
Sep 10 14:00:00.456789 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.456768    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:00:00.461283 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.461261    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"]
Sep 10 14:00:00.462397 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:00:00.462362    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446ce249_88a5_4a69_95ab_0e0da733553b.slice/crio-6a28c2ac16611b92d8a17251a3db8897ea39e8f062e19736d06aabb45868a069 WatchSource:0}: Error finding container 6a28c2ac16611b92d8a17251a3db8897ea39e8f062e19736d06aabb45868a069: Status 404 returned error can't find the container with id 6a28c2ac16611b92d8a17251a3db8897ea39e8f062e19736d06aabb45868a069
Sep 10 14:00:00.834062 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.834027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr" event={"ID":"446ce249-88a5-4a69-95ab-0e0da733553b","Type":"ContainerStarted","Data":"6a28c2ac16611b92d8a17251a3db8897ea39e8f062e19736d06aabb45868a069"}
Sep 10 14:00:00.835247 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:00.835214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp" event={"ID":"ad87b86c-7044-432a-8501-2e819ed59540","Type":"ContainerStarted","Data":"8cd2601f7aac6930360162907820af1a77613e50c879e1d43dd29e1748a89af6"}
Sep 10 14:00:03.136255 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:03.136224    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5_16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98/payload-processing/0.log"
Sep 10 14:00:04.862194 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:04.862158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr" event={"ID":"446ce249-88a5-4a69-95ab-0e0da733553b","Type":"ContainerStarted","Data":"f0bbdd18697f0fea4c915d067ad676f5e61c9746682a227520ae94ff6d2e92a0"}
Sep 10 14:00:04.863789 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:04.863765    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad87b86c-7044-432a-8501-2e819ed59540" containerID="50a35ff91ed6d60102193b14307c67cc45dd9d794f8c703e3a0c63defdcc124e" exitCode=0
Sep 10 14:00:04.863904 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:04.863815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp" event={"ID":"ad87b86c-7044-432a-8501-2e819ed59540","Type":"ContainerDied","Data":"50a35ff91ed6d60102193b14307c67cc45dd9d794f8c703e3a0c63defdcc124e"}
Sep 10 14:00:04.880073 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:04.880031    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr" podStartSLOduration=1.524263079 podStartE2EDuration="4.880018575s" totalImagesPullingTime="3.355755496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:04.879039361 +0000 UTC m=+1748.903854063" watchObservedRunningTime="2026-09-10 14:00:04.880018575 +0000 UTC m=+1748.904833278"
Sep 10 14:00:05.870829 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:05.870799    2577 generic.go:356] "Generic (PLEG): container finished" podID="446ce249-88a5-4a69-95ab-0e0da733553b" containerID="f0bbdd18697f0fea4c915d067ad676f5e61c9746682a227520ae94ff6d2e92a0" exitCode=0
Sep 10 14:00:05.871214 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:05.870854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr" event={"ID":"446ce249-88a5-4a69-95ab-0e0da733553b","Type":"ContainerDied","Data":"f0bbdd18697f0fea4c915d067ad676f5e61c9746682a227520ae94ff6d2e92a0"}
Sep 10 14:00:06.033692 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:06.033665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"
Sep 10 14:00:06.878006 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:06.877978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"
Sep 10 14:00:06.878374 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:06.878016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp" event={"ID":"ad87b86c-7044-432a-8501-2e819ed59540","Type":"ContainerDied","Data":"8cd2601f7aac6930360162907820af1a77613e50c879e1d43dd29e1748a89af6"}
Sep 10 14:00:06.878374 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:06.878064    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cd2601f7aac6930360162907820af1a77613e50c879e1d43dd29e1748a89af6"
Sep 10 14:00:07.019781 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:07.019752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"
Sep 10 14:00:07.885410 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:07.885384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"
Sep 10 14:00:07.885832 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:07.885425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr" event={"ID":"446ce249-88a5-4a69-95ab-0e0da733553b","Type":"ContainerDied","Data":"6a28c2ac16611b92d8a17251a3db8897ea39e8f062e19736d06aabb45868a069"}
Sep 10 14:00:07.885832 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:07.885471    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a28c2ac16611b92d8a17251a3db8897ea39e8f062e19736d06aabb45868a069"
Sep 10 14:00:12.037361 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.037328    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"]
Sep 10 14:00:12.037902 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.037884    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad87b86c-7044-432a-8501-2e819ed59540" containerName="cleanup"
Sep 10 14:00:12.037955 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.037905    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad87b86c-7044-432a-8501-2e819ed59540" containerName="cleanup"
Sep 10 14:00:12.037955 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.037932    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446ce249-88a5-4a69-95ab-0e0da733553b" containerName="cleanup"
Sep 10 14:00:12.037955 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.037938    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="446ce249-88a5-4a69-95ab-0e0da733553b" containerName="cleanup"
Sep 10 14:00:12.038057 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.038012    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad87b86c-7044-432a-8501-2e819ed59540" containerName="cleanup"
Sep 10 14:00:12.038057 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.038023    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="446ce249-88a5-4a69-95ab-0e0da733553b" containerName="cleanup"
Sep 10 14:00:12.040925 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.040909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.043506 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.043483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ae1e0d-dockercfg-rzk92\""
Sep 10 14:00:12.043655 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.043519    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ae1e0d\""
Sep 10 14:00:12.054411 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.054390    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"]
Sep 10 14:00:12.137461 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.137423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de21691-553a-4c6d-9e4a-3cc537286be4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.137668 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.137476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8tw\" (UniqueName: \"kubernetes.io/projected/0de21691-553a-4c6d-9e4a-3cc537286be4-kube-api-access-pp8tw\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.238513 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.238474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de21691-553a-4c6d-9e4a-3cc537286be4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.238689 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.238535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8tw\" (UniqueName: \"kubernetes.io/projected/0de21691-553a-4c6d-9e4a-3cc537286be4-kube-api-access-pp8tw\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.239222 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.239200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de21691-553a-4c6d-9e4a-3cc537286be4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.248743 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.248721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8tw\" (UniqueName: \"kubernetes.io/projected/0de21691-553a-4c6d-9e4a-3cc537286be4-kube-api-access-pp8tw\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.351910 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.351882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.629547 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.629505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"]
Sep 10 14:00:12.631086 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:00:12.631055    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de21691_553a_4c6d_9e4a_3cc537286be4.slice/crio-a1ab571604f22f523a8c9db0f093c7a7d225de2e7f2e53f6be48e2b2c0eff8b8 WatchSource:0}: Error finding container a1ab571604f22f523a8c9db0f093c7a7d225de2e7f2e53f6be48e2b2c0eff8b8: Status 404 returned error can't find the container with id a1ab571604f22f523a8c9db0f093c7a7d225de2e7f2e53f6be48e2b2c0eff8b8
Sep 10 14:00:12.913643 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.913557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh" event={"ID":"0de21691-553a-4c6d-9e4a-3cc537286be4","Type":"ContainerStarted","Data":"35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a"}
Sep 10 14:00:12.913643 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.913627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh" event={"ID":"0de21691-553a-4c6d-9e4a-3cc537286be4","Type":"ContainerStarted","Data":"a1ab571604f22f523a8c9db0f093c7a7d225de2e7f2e53f6be48e2b2c0eff8b8"}
Sep 10 14:00:12.913825 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.913715    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:12.949348 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:12.949295    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh" podStartSLOduration=0.949275106 podStartE2EDuration="949.275106ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:12.946870197 +0000 UTC m=+1756.971684916" watchObservedRunningTime="2026-09-10 14:00:12.949275106 +0000 UTC m=+1756.974089810"
Sep 10 14:00:23.920953 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:23.920920    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:49.423456 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.423243    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"]
Sep 10 14:00:49.423957 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.423487    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh" podUID="0de21691-553a-4c6d-9e4a-3cc537286be4" containerName="payload-pre-processing" containerID="cri-o://35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a" gracePeriod=30
Sep 10 14:00:49.698086 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.698052    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:49.710435 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.710395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp8tw\" (UniqueName: \"kubernetes.io/projected/0de21691-553a-4c6d-9e4a-3cc537286be4-kube-api-access-pp8tw\") pod \"0de21691-553a-4c6d-9e4a-3cc537286be4\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") "
Sep 10 14:00:49.710556 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.710483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de21691-553a-4c6d-9e4a-3cc537286be4-plugins-config-volume\") pod \"0de21691-553a-4c6d-9e4a-3cc537286be4\" (UID: \"0de21691-553a-4c6d-9e4a-3cc537286be4\") "
Sep 10 14:00:49.710840 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.710810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de21691-553a-4c6d-9e4a-3cc537286be4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0de21691-553a-4c6d-9e4a-3cc537286be4" (UID: "0de21691-553a-4c6d-9e4a-3cc537286be4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:00:49.710967 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.710903    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de21691-553a-4c6d-9e4a-3cc537286be4-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:00:49.712740 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.712709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de21691-553a-4c6d-9e4a-3cc537286be4-kube-api-access-pp8tw" (OuterVolumeSpecName: "kube-api-access-pp8tw") pod "0de21691-553a-4c6d-9e4a-3cc537286be4" (UID: "0de21691-553a-4c6d-9e4a-3cc537286be4"). InnerVolumeSpecName "kube-api-access-pp8tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:00:49.811621 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:49.811573    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8tw\" (UniqueName: \"kubernetes.io/projected/0de21691-553a-4c6d-9e4a-3cc537286be4-kube-api-access-pp8tw\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:00:50.122681 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.122648    2577 generic.go:356] "Generic (PLEG): container finished" podID="0de21691-553a-4c6d-9e4a-3cc537286be4" containerID="35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a" exitCode=0
Sep 10 14:00:50.122885 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.122710    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"
Sep 10 14:00:50.122885 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.122713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh" event={"ID":"0de21691-553a-4c6d-9e4a-3cc537286be4","Type":"ContainerDied","Data":"35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a"}
Sep 10 14:00:50.122885 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.122834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh" event={"ID":"0de21691-553a-4c6d-9e4a-3cc537286be4","Type":"ContainerDied","Data":"a1ab571604f22f523a8c9db0f093c7a7d225de2e7f2e53f6be48e2b2c0eff8b8"}
Sep 10 14:00:50.122885 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.122861    2577 scope.go:117] "RemoveContainer" containerID="35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a"
Sep 10 14:00:50.133212 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.133191    2577 scope.go:117] "RemoveContainer" containerID="35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a"
Sep 10 14:00:50.133497 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:00:50.133474    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a\": container with ID starting with 35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a not found: ID does not exist" containerID="35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a"
Sep 10 14:00:50.133562 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.133511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a"} err="failed to get container status \"35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a\": rpc error: code = NotFound desc = could not find container \"35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a\": container with ID starting with 35e99925d76334a29411257552a8e3cc60e89fd0f9bc5988997a88d627e3e69a not found: ID does not exist"
Sep 10 14:00:50.156328 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.156302    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"]
Sep 10 14:00:50.166379 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.166348    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae1e0d-b8ffbfbfd-brgwh"]
Sep 10 14:00:50.452995 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:50.452921    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de21691-553a-4c6d-9e4a-3cc537286be4" path="/var/lib/kubelet/pods/0de21691-553a-4c6d-9e4a-3cc537286be4/volumes"
Sep 10 14:00:56.606911 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:56.606886    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 14:00:56.608386 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:00:56.608364    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 14:01:17.360468 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.360432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"]
Sep 10 14:01:17.361022 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.360676    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5" podUID="16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" containerName="payload-processing" containerID="cri-o://2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88" gracePeriod=30
Sep 10 14:01:17.387001 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.386972    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"]
Sep 10 14:01:17.387286 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.387219    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2" podUID="56738747-e951-4863-b8c7-3afd798eed1e" containerName="payload-pre-processing" containerID="cri-o://289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8" gracePeriod=30
Sep 10 14:01:17.671967 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.669100    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 14:01:17.711256 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.711220    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 14:01:17.795074 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.795037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw6zk\" (UniqueName: \"kubernetes.io/projected/56738747-e951-4863-b8c7-3afd798eed1e-kube-api-access-zw6zk\") pod \"56738747-e951-4863-b8c7-3afd798eed1e\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") "
Sep 10 14:01:17.795262 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.795159    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-plugins-config-volume\") pod \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") "
Sep 10 14:01:17.795262 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.795203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxqpn\" (UniqueName: \"kubernetes.io/projected/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-kube-api-access-dxqpn\") pod \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\" (UID: \"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98\") "
Sep 10 14:01:17.795384 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.795284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56738747-e951-4863-b8c7-3afd798eed1e-plugins-config-volume\") pod \"56738747-e951-4863-b8c7-3afd798eed1e\" (UID: \"56738747-e951-4863-b8c7-3afd798eed1e\") "
Sep 10 14:01:17.795706 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.795678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56738747-e951-4863-b8c7-3afd798eed1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56738747-e951-4863-b8c7-3afd798eed1e" (UID: "56738747-e951-4863-b8c7-3afd798eed1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:01:17.795706 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.795688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" (UID: "16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:01:17.797333 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.797316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-kube-api-access-dxqpn" (OuterVolumeSpecName: "kube-api-access-dxqpn") pod "16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" (UID: "16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98"). InnerVolumeSpecName "kube-api-access-dxqpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:01:17.797390 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.797374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56738747-e951-4863-b8c7-3afd798eed1e-kube-api-access-zw6zk" (OuterVolumeSpecName: "kube-api-access-zw6zk") pod "56738747-e951-4863-b8c7-3afd798eed1e" (UID: "56738747-e951-4863-b8c7-3afd798eed1e"). InnerVolumeSpecName "kube-api-access-zw6zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:01:17.896712 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.896625    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56738747-e951-4863-b8c7-3afd798eed1e-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:01:17.896712 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.896657    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6zk\" (UniqueName: \"kubernetes.io/projected/56738747-e951-4863-b8c7-3afd798eed1e-kube-api-access-zw6zk\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:01:17.896712 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.896669    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:01:17.896712 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:17.896678    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxqpn\" (UniqueName: \"kubernetes.io/projected/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98-kube-api-access-dxqpn\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:01:18.288964 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.288871    2577 generic.go:356] "Generic (PLEG): container finished" podID="16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" containerID="2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88" exitCode=0
Sep 10 14:01:18.288964 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.288919    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"
Sep 10 14:01:18.289187 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.288964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5" event={"ID":"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98","Type":"ContainerDied","Data":"2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88"}
Sep 10 14:01:18.289187 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.289016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5" event={"ID":"16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98","Type":"ContainerDied","Data":"e641cb7b96ff0fbe6b73e75fd1294c7026bcf9ac5e7601529678a19080a0d9ef"}
Sep 10 14:01:18.289187 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.289050    2577 scope.go:117] "RemoveContainer" containerID="2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88"
Sep 10 14:01:18.290477 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.290450    2577 generic.go:356] "Generic (PLEG): container finished" podID="56738747-e951-4863-b8c7-3afd798eed1e" containerID="289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8" exitCode=0
Sep 10 14:01:18.290622 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.290483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2" event={"ID":"56738747-e951-4863-b8c7-3afd798eed1e","Type":"ContainerDied","Data":"289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8"}
Sep 10 14:01:18.290622 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.290535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2" event={"ID":"56738747-e951-4863-b8c7-3afd798eed1e","Type":"ContainerDied","Data":"204af7d84985bbe60260735646624836d163d6d16c5abef3c22580aa2acea8a5"}
Sep 10 14:01:18.290622 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.290498    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"
Sep 10 14:01:18.302535 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.302508    2577 scope.go:117] "RemoveContainer" containerID="2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88"
Sep 10 14:01:18.302927 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:01:18.302901    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88\": container with ID starting with 2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88 not found: ID does not exist" containerID="2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88"
Sep 10 14:01:18.303033 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.302937    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88"} err="failed to get container status \"2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88\": rpc error: code = NotFound desc = could not find container \"2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88\": container with ID starting with 2b27738be2aab2f875e39f64fecddec104c17ec1b87549e3e2a2c8de6d3a8c88 not found: ID does not exist"
Sep 10 14:01:18.303033 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.302955    2577 scope.go:117] "RemoveContainer" containerID="289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8"
Sep 10 14:01:18.312588 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.312557    2577 scope.go:117] "RemoveContainer" containerID="289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8"
Sep 10 14:01:18.312914 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:01:18.312896    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8\": container with ID starting with 289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8 not found: ID does not exist" containerID="289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8"
Sep 10 14:01:18.312979 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.312922    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8"} err="failed to get container status \"289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8\": rpc error: code = NotFound desc = could not find container \"289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8\": container with ID starting with 289e43c3bad5de2d55dfde2e8e0d5f301939ed16f7a24206bbb641efcdd6ecd8 not found: ID does not exist"
Sep 10 14:01:18.340866 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.340823    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"]
Sep 10 14:01:18.351434 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.351401    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fe383-78c6c8b946-lj4c2"]
Sep 10 14:01:18.369833 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.369800    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"]
Sep 10 14:01:18.372633 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.372609    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fe383-6f48564747-jgtv5"]
Sep 10 14:01:18.452989 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.452945    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" path="/var/lib/kubelet/pods/16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98/volumes"
Sep 10 14:01:18.453423 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:01:18.453404    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56738747-e951-4863-b8c7-3afd798eed1e" path="/var/lib/kubelet/pods/56738747-e951-4863-b8c7-3afd798eed1e/volumes"
Sep 10 14:02:15.950324 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:15.950289    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"]
Sep 10 14:02:15.950800 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:15.950526    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy" containerID="cri-o://b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b" gracePeriod=30
Sep 10 14:02:21.211117 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.211092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 14:02:21.349841 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.349794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-podinfo\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350046 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.349885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-certs\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350046 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.349993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-credential-socket\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350046 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-socket\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350283 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7xft\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-kube-api-access-r7xft\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350283 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350136    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:21.350283 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-envoy\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350283 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-data\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350283 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350234    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:21.350539 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:21.350539 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c592721-527b-4db9-8b60-dc79ed27cb79-istiod-ca-cert\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350539 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-token\") pod \"4c592721-527b-4db9-8b60-dc79ed27cb79\" (UID: \"4c592721-527b-4db9-8b60-dc79ed27cb79\") "
Sep 10 14:02:21.350690 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350546    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:21.350690 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c592721-527b-4db9-8b60-dc79ed27cb79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:02:21.350845 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350825    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.350889 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350852    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.350889 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350870    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.350889 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350884    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.351011 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.350897    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c592721-527b-4db9-8b60-dc79ed27cb79-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.352090 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.352070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:02:21.352369 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.352342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:02:21.352732 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.352702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:21.352732 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.352707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-kube-api-access-r7xft" (OuterVolumeSpecName: "kube-api-access-r7xft") pod "4c592721-527b-4db9-8b60-dc79ed27cb79" (UID: "4c592721-527b-4db9-8b60-dc79ed27cb79"). InnerVolumeSpecName "kube-api-access-r7xft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:02:21.451612 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.451543    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7xft\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-kube-api-access-r7xft\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.451612 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.451606    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.451612 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.451621    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.451857 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.451633    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c592721-527b-4db9-8b60-dc79ed27cb79-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:21.649467 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.649371    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerID="b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b" exitCode=0
Sep 10 14:02:21.649467 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.649429    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"
Sep 10 14:02:21.649467 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.649449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" event={"ID":"4c592721-527b-4db9-8b60-dc79ed27cb79","Type":"ContainerDied","Data":"b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b"}
Sep 10 14:02:21.649776 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.649517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr" event={"ID":"4c592721-527b-4db9-8b60-dc79ed27cb79","Type":"ContainerDied","Data":"b5c5ccf7664c0984c0b27146ef4ded555063efd57223cc5d1df7d23c902861c2"}
Sep 10 14:02:21.649776 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.649548    2577 scope.go:117] "RemoveContainer" containerID="b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b"
Sep 10 14:02:21.659022 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.659003    2577 scope.go:117] "RemoveContainer" containerID="7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"
Sep 10 14:02:21.667509 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.667489    2577 scope.go:117] "RemoveContainer" containerID="b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b"
Sep 10 14:02:21.667850 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:02:21.667828    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b\": container with ID starting with b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b not found: ID does not exist" containerID="b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b"
Sep 10 14:02:21.667917 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.667857    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b"} err="failed to get container status \"b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b\": rpc error: code = NotFound desc = could not find container \"b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b\": container with ID starting with b799213c1613622016980ffe3f066211c7c7945d6294ff598bb23fa974a5917b not found: ID does not exist"
Sep 10 14:02:21.667917 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.667875    2577 scope.go:117] "RemoveContainer" containerID="7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"
Sep 10 14:02:21.668140 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:02:21.668122    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b\": container with ID starting with 7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b not found: ID does not exist" containerID="7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"
Sep 10 14:02:21.668179 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.668146    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b"} err="failed to get container status \"7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b\": rpc error: code = NotFound desc = could not find container \"7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b\": container with ID starting with 7b62f10d4d05ebad601c019a824aaf6946dc064e4b512cefee505321d4353f8b not found: ID does not exist"
Sep 10 14:02:21.679281 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.679257    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"]
Sep 10 14:02:21.694513 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:21.694487    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fe383-openshift-default-5789b5d97b-dv9cr"]
Sep 10 14:02:22.454126 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:22.454092    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" path="/var/lib/kubelet/pods/4c592721-527b-4db9-8b60-dc79ed27cb79/volumes"
Sep 10 14:02:35.527842 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:35.527802    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"]
Sep 10 14:02:35.528442 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:35.528014    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy" containerID="cri-o://f5d88855d40505516efd5bba1f8944c6abe2168979a5a874df0e2d94f7614d11" gracePeriod=30
Sep 10 14:02:40.756178 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.756144    2577 generic.go:356] "Generic (PLEG): container finished" podID="a359b055-9710-487b-a96a-69d6f91682e1" containerID="f5d88855d40505516efd5bba1f8944c6abe2168979a5a874df0e2d94f7614d11" exitCode=0
Sep 10 14:02:40.756541 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.756214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" event={"ID":"a359b055-9710-487b-a96a-69d6f91682e1","Type":"ContainerDied","Data":"f5d88855d40505516efd5bba1f8944c6abe2168979a5a874df0e2d94f7614d11"}
Sep 10 14:02:40.756541 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.756282    2577 scope.go:117] "RemoveContainer" containerID="afba5489d8b3d6f70f6dfa5ada17384f26d8636bdc9e30f93a66bf7d5eb0e82a"
Sep 10 14:02:40.785158 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.785137    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 14:02:40.839069 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln47l\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-kube-api-access-ln47l\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839206 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-credential-socket\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839206 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-istio-token\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839330 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a359b055-9710-487b-a96a-69d6f91682e1-istio-podinfo\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839330 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-socket\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839330 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-certs\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839492 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a359b055-9710-487b-a96a-69d6f91682e1-istiod-ca-cert\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839492 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-data\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839492 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839457    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-envoy\") pod \"a359b055-9710-487b-a96a-69d6f91682e1\" (UID: \"a359b055-9710-487b-a96a-69d6f91682e1\") "
Sep 10 14:02:40.839689 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839492    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:40.839689 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839533    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:40.839689 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:40.839837 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:40.839837 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a359b055-9710-487b-a96a-69d6f91682e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:02:40.839946 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839928    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.839992 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839953    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.839992 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839969    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.839992 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839982    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a359b055-9710-487b-a96a-69d6f91682e1-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.840105 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.839994    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.841310 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.841290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a359b055-9710-487b-a96a-69d6f91682e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:02:40.841739 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.841720    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:02:40.841811 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.841748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-kube-api-access-ln47l" (OuterVolumeSpecName: "kube-api-access-ln47l") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "kube-api-access-ln47l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:02:40.841811 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.841773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a359b055-9710-487b-a96a-69d6f91682e1" (UID: "a359b055-9710-487b-a96a-69d6f91682e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:40.941234 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.941207    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a359b055-9710-487b-a96a-69d6f91682e1-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.941234 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.941231    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln47l\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-kube-api-access-ln47l\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.941363 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.941241    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a359b055-9710-487b-a96a-69d6f91682e1-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:40.941363 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:40.941250    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a359b055-9710-487b-a96a-69d6f91682e1-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:02:41.764560 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:41.764526    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"
Sep 10 14:02:41.764560 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:41.764558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf" event={"ID":"a359b055-9710-487b-a96a-69d6f91682e1","Type":"ContainerDied","Data":"46f8816d77c896b6fd3fce60ce9410f8458c07963b12d598b204d2e863fec386"}
Sep 10 14:02:41.765077 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:41.764632    2577 scope.go:117] "RemoveContainer" containerID="f5d88855d40505516efd5bba1f8944c6abe2168979a5a874df0e2d94f7614d11"
Sep 10 14:02:41.818632 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:41.818607    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"]
Sep 10 14:02:41.825079 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:41.825055    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc6ea5-openshift-default-57874f989b-66sgf"]
Sep 10 14:02:42.452980 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:42.452947    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a359b055-9710-487b-a96a-69d6f91682e1" path="/var/lib/kubelet/pods/a359b055-9710-487b-a96a-69d6f91682e1/volumes"
Sep 10 14:02:46.586859 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:46.586820    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"]
Sep 10 14:02:46.589043 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:46.589021    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-579ae1-29817480-ssnvp"]
Sep 10 14:02:48.453102 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:02:48.453058    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad87b86c-7044-432a-8501-2e819ed59540" path="/var/lib/kubelet/pods/ad87b86c-7044-432a-8501-2e819ed59540/volumes"
Sep 10 14:03:06.253563 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.253525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"]
Sep 10 14:03:06.256566 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.256537    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5c012-29817480-8g6lr"]
Sep 10 14:03:06.335559 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.335530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"]
Sep 10 14:03:06.335813 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.335773    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6" podUID="371d066f-8f92-425c-8e89-7151b3837c4c" containerName="payload-pre-processing" containerID="cri-o://fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7" gracePeriod=30
Sep 10 14:03:06.454095 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.454069    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446ce249-88a5-4a69-95ab-0e0da733553b" path="/var/lib/kubelet/pods/446ce249-88a5-4a69-95ab-0e0da733553b/volumes"
Sep 10 14:03:06.585159 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.585140    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 14:03:06.662607 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.662568    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d066f-8f92-425c-8e89-7151b3837c4c-plugins-config-volume\") pod \"371d066f-8f92-425c-8e89-7151b3837c4c\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") "
Sep 10 14:03:06.662817 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.662617    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/371d066f-8f92-425c-8e89-7151b3837c4c-kube-api-access-54ggj\") pod \"371d066f-8f92-425c-8e89-7151b3837c4c\" (UID: \"371d066f-8f92-425c-8e89-7151b3837c4c\") "
Sep 10 14:03:06.662949 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.662925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371d066f-8f92-425c-8e89-7151b3837c4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "371d066f-8f92-425c-8e89-7151b3837c4c" (UID: "371d066f-8f92-425c-8e89-7151b3837c4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:03:06.664561 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.664534    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d066f-8f92-425c-8e89-7151b3837c4c-kube-api-access-54ggj" (OuterVolumeSpecName: "kube-api-access-54ggj") pod "371d066f-8f92-425c-8e89-7151b3837c4c" (UID: "371d066f-8f92-425c-8e89-7151b3837c4c"). InnerVolumeSpecName "kube-api-access-54ggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:06.764423 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.764387    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d066f-8f92-425c-8e89-7151b3837c4c-plugins-config-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:06.764423 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.764417    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/371d066f-8f92-425c-8e89-7151b3837c4c-kube-api-access-54ggj\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:06.899250 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.899215    2577 generic.go:356] "Generic (PLEG): container finished" podID="371d066f-8f92-425c-8e89-7151b3837c4c" containerID="fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7" exitCode=0
Sep 10 14:03:06.899408 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.899257    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"
Sep 10 14:03:06.899408 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.899257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6" event={"ID":"371d066f-8f92-425c-8e89-7151b3837c4c","Type":"ContainerDied","Data":"fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7"}
Sep 10 14:03:06.899408 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.899375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6" event={"ID":"371d066f-8f92-425c-8e89-7151b3837c4c","Type":"ContainerDied","Data":"2c8c52b86471102daea5c931fee6c15d78b519db34cf9280ec9b96caa39c181e"}
Sep 10 14:03:06.899408 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.899406    2577 scope.go:117] "RemoveContainer" containerID="fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7"
Sep 10 14:03:06.908940 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.908920    2577 scope.go:117] "RemoveContainer" containerID="fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7"
Sep 10 14:03:06.909199 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:03:06.909181    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7\": container with ID starting with fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7 not found: ID does not exist" containerID="fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7"
Sep 10 14:03:06.909243 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.909208    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7"} err="failed to get container status \"fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7\": rpc error: code = NotFound desc = could not find container \"fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7\": container with ID starting with fa1cde9fb621c875171f7da2dbac04f9dbbad84f5adba4164b18243cf149bff7 not found: ID does not exist"
Sep 10 14:03:06.938092 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.938069    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"]
Sep 10 14:03:06.947162 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:06.947134    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5c012-86649856bb-4fvb6"]
Sep 10 14:03:08.452245 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:08.452209    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371d066f-8f92-425c-8e89-7151b3837c4c" path="/var/lib/kubelet/pods/371d066f-8f92-425c-8e89-7151b3837c4c/volumes"
Sep 10 14:03:16.751298 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:16.751270    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"]
Sep 10 14:03:16.751688 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:16.751461    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" containerID="cri-o://0a97b351b09fc953ff6f0cdbbe69ff02a15a362e8434b8bb88a56aebb4537c47" gracePeriod=30
Sep 10 14:03:19.112439 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.112399    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 14:03:19.112828 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.112475    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 14:03:19.688441 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.688402    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9955587-k9ctw"]
Sep 10 14:03:19.688654 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.688630    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9955587-k9ctw" podUID="31d93039-d87a-4620-96ec-8ba8eac4abe5" containerName="manager" containerID="cri-o://2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd" gracePeriod=30
Sep 10 14:03:19.927826 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.927796    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 14:03:19.971353 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.971283    2577 generic.go:356] "Generic (PLEG): container finished" podID="31d93039-d87a-4620-96ec-8ba8eac4abe5" containerID="2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd" exitCode=0
Sep 10 14:03:19.971353 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.971328    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-k9ctw"
Sep 10 14:03:19.971535 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.971362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-k9ctw" event={"ID":"31d93039-d87a-4620-96ec-8ba8eac4abe5","Type":"ContainerDied","Data":"2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd"}
Sep 10 14:03:19.971535 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.971416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-k9ctw" event={"ID":"31d93039-d87a-4620-96ec-8ba8eac4abe5","Type":"ContainerDied","Data":"6819b3436f6cfb7a116a9180b02681048ed4cc925deb7e9fea37dc135958aa26"}
Sep 10 14:03:19.971535 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.971443    2577 scope.go:117] "RemoveContainer" containerID="2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd"
Sep 10 14:03:19.981270 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.981253    2577 scope.go:117] "RemoveContainer" containerID="2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd"
Sep 10 14:03:19.981539 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:03:19.981520    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd\": container with ID starting with 2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd not found: ID does not exist" containerID="2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd"
Sep 10 14:03:19.981629 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:19.981547    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd"} err="failed to get container status \"2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd\": rpc error: code = NotFound desc = could not find container \"2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd\": container with ID starting with 2dbc29ac8bfbc978affec00e15feaba3c39ec18bb41398753a5e229cf789f5fd not found: ID does not exist"
Sep 10 14:03:20.081398 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.081373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-metrics-certs\") pod \"31d93039-d87a-4620-96ec-8ba8eac4abe5\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") "
Sep 10 14:03:20.081517 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.081440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-webhook-cert\") pod \"31d93039-d87a-4620-96ec-8ba8eac4abe5\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") "
Sep 10 14:03:20.081606 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.081517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzl8b\" (UniqueName: \"kubernetes.io/projected/31d93039-d87a-4620-96ec-8ba8eac4abe5-kube-api-access-tzl8b\") pod \"31d93039-d87a-4620-96ec-8ba8eac4abe5\" (UID: \"31d93039-d87a-4620-96ec-8ba8eac4abe5\") "
Sep 10 14:03:20.083499 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.083474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "31d93039-d87a-4620-96ec-8ba8eac4abe5" (UID: "31d93039-d87a-4620-96ec-8ba8eac4abe5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:03:20.083649 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.083490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "31d93039-d87a-4620-96ec-8ba8eac4abe5" (UID: "31d93039-d87a-4620-96ec-8ba8eac4abe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:03:20.083649 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.083501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d93039-d87a-4620-96ec-8ba8eac4abe5-kube-api-access-tzl8b" (OuterVolumeSpecName: "kube-api-access-tzl8b") pod "31d93039-d87a-4620-96ec-8ba8eac4abe5" (UID: "31d93039-d87a-4620-96ec-8ba8eac4abe5"). InnerVolumeSpecName "kube-api-access-tzl8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:20.183068 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.183042    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzl8b\" (UniqueName: \"kubernetes.io/projected/31d93039-d87a-4620-96ec-8ba8eac4abe5-kube-api-access-tzl8b\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:20.183068 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.183067    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:20.183413 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.183077    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d93039-d87a-4620-96ec-8ba8eac4abe5-webhook-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:20.302751 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.302716    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9955587-k9ctw"]
Sep 10 14:03:20.305482 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.305455    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9955587-k9ctw"]
Sep 10 14:03:20.452347 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:20.452315    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d93039-d87a-4620-96ec-8ba8eac4abe5" path="/var/lib/kubelet/pods/31d93039-d87a-4620-96ec-8ba8eac4abe5/volumes"
Sep 10 14:03:21.985314 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.985285    2577 generic.go:356] "Generic (PLEG): container finished" podID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerID="0a97b351b09fc953ff6f0cdbbe69ff02a15a362e8434b8bb88a56aebb4537c47" exitCode=0
Sep 10 14:03:21.985667 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.985345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" event={"ID":"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd","Type":"ContainerDied","Data":"0a97b351b09fc953ff6f0cdbbe69ff02a15a362e8434b8bb88a56aebb4537c47"}
Sep 10 14:03:21.985667 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.985399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7" event={"ID":"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd","Type":"ContainerDied","Data":"256aa53f75d6931992e0abf176a0d4f4e3282d68c5e970b2ccf2b209a9fd5da6"}
Sep 10 14:03:21.985667 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.985416    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256aa53f75d6931992e0abf176a0d4f4e3282d68c5e970b2ccf2b209a9fd5da6"
Sep 10 14:03:21.985667 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.985431    2577 scope.go:117] "RemoveContainer" containerID="ccb5b864721d94c0118c1ea54447389c663d4ae7f64a6dd0c601f9d9f11d8390"
Sep 10 14:03:21.990068 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.990047    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 14:03:21.999074 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999058    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-data\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999144 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-podinfo\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999144 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999113    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-envoy\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999144 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-certs\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999280 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999202    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istiod-ca-cert\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999280 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-credential-socket\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999389 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:21.999389 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvt6h\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-kube-api-access-mvt6h\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999491 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-token\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999491 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-socket\") pod \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\" (UID: \"62d18c6c-53db-4b10-b6a9-9bedf83ec7fd\") "
Sep 10 14:03:21.999491 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:21.999649 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:03:21.999768 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:21.999836 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:21.999949 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999928    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:21.999994 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999954    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-data\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:21.999994 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999969    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-workload-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:21.999994 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999983    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istiod-ca-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.000097 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:21.999995    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-credential-socket\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.001177 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.001146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:03:22.001389 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.001362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-kube-api-access-mvt6h" (OuterVolumeSpecName: "kube-api-access-mvt6h") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "kube-api-access-mvt6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:22.001539 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.001519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:22.001659 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.001637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" (UID: "62d18c6c-53db-4b10-b6a9-9bedf83ec7fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:22.100356 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.100332    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvt6h\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-kube-api-access-mvt6h\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.100356 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.100354    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-token\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.100506 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.100364    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-podinfo\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.100506 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.100372    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd-istio-envoy\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.991478 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:22.991451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"
Sep 10 14:03:23.046335 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:23.046302    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"]
Sep 10 14:03:23.049286 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:23.049260    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5c012-openshift-default-7b8c88d4dd-cdbx7"]
Sep 10 14:03:24.452022 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:24.451987    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" path="/var/lib/kubelet/pods/62d18c6c-53db-4b10-b6a9-9bedf83ec7fd/volumes"
Sep 10 14:03:56.774305 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:03:56.774276    2577 scope.go:117] "RemoveContainer" containerID="0a97b351b09fc953ff6f0cdbbe69ff02a15a362e8434b8bb88a56aebb4537c47"
Sep 10 14:04:21.165115 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165084    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9955587-s9bhj"]
Sep 10 14:04:21.165749 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165728    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165751    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165767    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d93039-d87a-4620-96ec-8ba8eac4abe5" containerName="manager"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165776    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d93039-d87a-4620-96ec-8ba8eac4abe5" containerName="manager"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165786    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165794    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165808    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165816    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165837    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de21691-553a-4c6d-9e4a-3cc537286be4" containerName="payload-pre-processing"
Sep 10 14:04:21.165846 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165846    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de21691-553a-4c6d-9e4a-3cc537286be4" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165861    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56738747-e951-4863-b8c7-3afd798eed1e" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165870    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="56738747-e951-4863-b8c7-3afd798eed1e" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165888    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d066f-8f92-425c-8e89-7151b3837c4c" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165896    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d066f-8f92-425c-8e89-7151b3837c4c" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165913    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" containerName="payload-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.165921    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" containerName="payload-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166024    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de21691-553a-4c6d-9e4a-3cc537286be4" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166037    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166045    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d93039-d87a-4620-96ec-8ba8eac4abe5" containerName="manager"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166054    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166064    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166074    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="56738747-e951-4863-b8c7-3afd798eed1e" containerName="payload-pre-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166082    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c7c51b-6a9c-4a24-9b4b-0c5c8ec0bb98" containerName="payload-processing"
Sep 10 14:04:21.166558 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.166090    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d066f-8f92-425c-8e89-7151b3837c4c" containerName="payload-pre-processing"
Sep 10 14:04:21.169553 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.169534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.173210 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.173189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 14:04:21.173305 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.173204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 14:04:21.173305 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.173246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lz67j\""
Sep 10 14:04:21.173428 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.173340    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 14:04:21.181462 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.181438    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9955587-s9bhj"]
Sep 10 14:04:21.202342 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.202310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-metrics-certs\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.202456 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.202421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb2c\" (UniqueName: \"kubernetes.io/projected/768e4c2e-6d4d-4eae-9261-2d670de128f8-kube-api-access-cxb2c\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.202497 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.202474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-webhook-cert\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.303779 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.303749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-metrics-certs\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.303919 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.303809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb2c\" (UniqueName: \"kubernetes.io/projected/768e4c2e-6d4d-4eae-9261-2d670de128f8-kube-api-access-cxb2c\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.303919 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.303848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-webhook-cert\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.306280 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.306257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-webhook-cert\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.306437 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.306351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-metrics-certs\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.313903 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.313878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb2c\" (UniqueName: \"kubernetes.io/projected/768e4c2e-6d4d-4eae-9261-2d670de128f8-kube-api-access-cxb2c\") pod \"maas-controller-5b9955587-s9bhj\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") " pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.480812 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.480728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:21.764090 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:21.764063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9955587-s9bhj"]
Sep 10 14:04:21.766129 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:04:21.766103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768e4c2e_6d4d_4eae_9261_2d670de128f8.slice/crio-b4647accd52f95f3a6c99f5a561b4183ca78b5f8b8e0bb8884105f995db0ab3a WatchSource:0}: Error finding container b4647accd52f95f3a6c99f5a561b4183ca78b5f8b8e0bb8884105f995db0ab3a: Status 404 returned error can't find the container with id b4647accd52f95f3a6c99f5a561b4183ca78b5f8b8e0bb8884105f995db0ab3a
Sep 10 14:04:22.301691 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:22.301645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-s9bhj" event={"ID":"768e4c2e-6d4d-4eae-9261-2d670de128f8","Type":"ContainerStarted","Data":"b4647accd52f95f3a6c99f5a561b4183ca78b5f8b8e0bb8884105f995db0ab3a"}
Sep 10 14:04:23.308905 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:23.308864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-s9bhj" event={"ID":"768e4c2e-6d4d-4eae-9261-2d670de128f8","Type":"ContainerStarted","Data":"992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94"}
Sep 10 14:04:23.309383 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:23.309020    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:23.332905 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:23.332849    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9955587-s9bhj" podStartSLOduration=1.5583408090000002 podStartE2EDuration="2.33283245s" totalImagesPullingTime="774.491641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:04:23.33118075 +0000 UTC m=+2007.355995454" watchObservedRunningTime="2026-09-10 14:04:23.33283245 +0000 UTC m=+2007.357647153"
Sep 10 14:04:34.316648 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:34.316617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:43.753365 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:43.753328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9955587-s9bhj"]
Sep 10 14:04:43.753858 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:43.753550    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9955587-s9bhj" podUID="768e4c2e-6d4d-4eae-9261-2d670de128f8" containerName="manager" containerID="cri-o://992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94" gracePeriod=30
Sep 10 14:04:44.002908 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.002881    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:44.129149 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.129111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-metrics-certs\") pod \"768e4c2e-6d4d-4eae-9261-2d670de128f8\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") "
Sep 10 14:04:44.129334 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.129161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxb2c\" (UniqueName: \"kubernetes.io/projected/768e4c2e-6d4d-4eae-9261-2d670de128f8-kube-api-access-cxb2c\") pod \"768e4c2e-6d4d-4eae-9261-2d670de128f8\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") "
Sep 10 14:04:44.129334 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.129194    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-webhook-cert\") pod \"768e4c2e-6d4d-4eae-9261-2d670de128f8\" (UID: \"768e4c2e-6d4d-4eae-9261-2d670de128f8\") "
Sep 10 14:04:44.131353 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.131315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "768e4c2e-6d4d-4eae-9261-2d670de128f8" (UID: "768e4c2e-6d4d-4eae-9261-2d670de128f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:04:44.131474 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.131396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "768e4c2e-6d4d-4eae-9261-2d670de128f8" (UID: "768e4c2e-6d4d-4eae-9261-2d670de128f8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:04:44.131474 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.131398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768e4c2e-6d4d-4eae-9261-2d670de128f8-kube-api-access-cxb2c" (OuterVolumeSpecName: "kube-api-access-cxb2c") pod "768e4c2e-6d4d-4eae-9261-2d670de128f8" (UID: "768e4c2e-6d4d-4eae-9261-2d670de128f8"). InnerVolumeSpecName "kube-api-access-cxb2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:04:44.230833 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.230797    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-webhook-cert\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:04:44.230833 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.230825    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768e4c2e-6d4d-4eae-9261-2d670de128f8-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:04:44.231011 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.230847    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb2c\" (UniqueName: \"kubernetes.io/projected/768e4c2e-6d4d-4eae-9261-2d670de128f8-kube-api-access-cxb2c\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:04:44.416703 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.416615    2577 generic.go:356] "Generic (PLEG): container finished" podID="768e4c2e-6d4d-4eae-9261-2d670de128f8" containerID="992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94" exitCode=0
Sep 10 14:04:44.416703 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.416666    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-s9bhj"
Sep 10 14:04:44.416889 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.416699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-s9bhj" event={"ID":"768e4c2e-6d4d-4eae-9261-2d670de128f8","Type":"ContainerDied","Data":"992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94"}
Sep 10 14:04:44.416889 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.416758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-s9bhj" event={"ID":"768e4c2e-6d4d-4eae-9261-2d670de128f8","Type":"ContainerDied","Data":"b4647accd52f95f3a6c99f5a561b4183ca78b5f8b8e0bb8884105f995db0ab3a"}
Sep 10 14:04:44.416889 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.416782    2577 scope.go:117] "RemoveContainer" containerID="992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94"
Sep 10 14:04:44.427517 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.427496    2577 scope.go:117] "RemoveContainer" containerID="992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94"
Sep 10 14:04:44.427832 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:04:44.427809    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94\": container with ID starting with 992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94 not found: ID does not exist" containerID="992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94"
Sep 10 14:04:44.427911 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.427846    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94"} err="failed to get container status \"992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94\": rpc error: code = NotFound desc = could not find container \"992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94\": container with ID starting with 992d2640e1cf1343dd2dca8a3d61047ba6a00f91c989841ba44c350f942b9c94 not found: ID does not exist"
Sep 10 14:04:44.463393 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.463366    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9955587-s9bhj"]
Sep 10 14:04:44.465038 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:44.465020    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9955587-s9bhj"]
Sep 10 14:04:45.272762 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.272725    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9955587-6r22m"]
Sep 10 14:04:45.273135 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273128    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy"
Sep 10 14:04:45.273175 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273139    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy"
Sep 10 14:04:45.273175 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273151    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy"
Sep 10 14:04:45.273175 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273157    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy"
Sep 10 14:04:45.273175 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273165    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy"
Sep 10 14:04:45.273175 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273172    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy"
Sep 10 14:04:45.273331 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273183    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768e4c2e-6d4d-4eae-9261-2d670de128f8" containerName="manager"
Sep 10 14:04:45.273331 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273188    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="768e4c2e-6d4d-4eae-9261-2d670de128f8" containerName="manager"
Sep 10 14:04:45.273331 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273259    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c592721-527b-4db9-8b60-dc79ed27cb79" containerName="istio-proxy"
Sep 10 14:04:45.273331 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273271    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a359b055-9710-487b-a96a-69d6f91682e1" containerName="istio-proxy"
Sep 10 14:04:45.273331 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273277    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="768e4c2e-6d4d-4eae-9261-2d670de128f8" containerName="manager"
Sep 10 14:04:45.273331 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.273284    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d18c6c-53db-4b10-b6a9-9bedf83ec7fd" containerName="istio-proxy"
Sep 10 14:04:45.277750 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.277725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.281414 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.281388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lz67j\""
Sep 10 14:04:45.281614 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.281389    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 14:04:45.281614 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.281530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 14:04:45.281744 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.281572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 14:04:45.286878 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.286853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9955587-6r22m"]
Sep 10 14:04:45.445215 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.445182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-metrics-certs\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.445390 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.445230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572dd\" (UniqueName: \"kubernetes.io/projected/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-kube-api-access-572dd\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.445390 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.445345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-webhook-cert\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.546446 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.546346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-webhook-cert\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.546446 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.546448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-metrics-certs\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.546722 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.546469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572dd\" (UniqueName: \"kubernetes.io/projected/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-kube-api-access-572dd\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.549057 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.549033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-webhook-cert\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.549160 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.549033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-metrics-certs\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.557023 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.556999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572dd\" (UniqueName: \"kubernetes.io/projected/5b6b22ad-31e9-4a3e-8648-7adcf17e1b10-kube-api-access-572dd\") pod \"maas-controller-5b9955587-6r22m\" (UID: \"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10\") " pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.589991 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.589958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:45.873761 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:45.873731    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9955587-6r22m"]
Sep 10 14:04:45.875807 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:04:45.875776    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6b22ad_31e9_4a3e_8648_7adcf17e1b10.slice/crio-8c9ff172055073f93c50228ad934ce7e920ba12377cf0736b01a66cfb6a338ac WatchSource:0}: Error finding container 8c9ff172055073f93c50228ad934ce7e920ba12377cf0736b01a66cfb6a338ac: Status 404 returned error can't find the container with id 8c9ff172055073f93c50228ad934ce7e920ba12377cf0736b01a66cfb6a338ac
Sep 10 14:04:46.436695 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:46.436652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-6r22m" event={"ID":"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10","Type":"ContainerStarted","Data":"35893f5cf8186bcc8a32ca4c9c5d3ad26e08f5ddc85d9942bd1a5ed53d212ac5"}
Sep 10 14:04:46.437145 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:46.436718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:04:46.437145 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:46.436739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9955587-6r22m" event={"ID":"5b6b22ad-31e9-4a3e-8648-7adcf17e1b10","Type":"ContainerStarted","Data":"8c9ff172055073f93c50228ad934ce7e920ba12377cf0736b01a66cfb6a338ac"}
Sep 10 14:04:46.452956 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:46.452918    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768e4c2e-6d4d-4eae-9261-2d670de128f8" path="/var/lib/kubelet/pods/768e4c2e-6d4d-4eae-9261-2d670de128f8/volumes"
Sep 10 14:04:46.476060 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:46.475997    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9955587-6r22m" podStartSLOduration=1.1226874470000001 podStartE2EDuration="1.475978453s" totalImagesPullingTime="353.291006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:04:46.475596795 +0000 UTC m=+2030.500411489" watchObservedRunningTime="2026-09-10 14:04:46.475978453 +0000 UTC m=+2030.500793178"
Sep 10 14:04:57.444820 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:04:57.444789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9955587-6r22m"
Sep 10 14:05:49.495678 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:49.495640    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"]
Sep 10 14:05:49.496130 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:49.495822    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" podUID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" containerName="manager" containerID="cri-o://42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413" gracePeriod=10
Sep 10 14:05:51.648783 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.648733    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" podUID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": dial tcp 10.134.0.34:8081: connect: connection refused"
Sep 10 14:05:51.748684 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.748656    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 14:05:51.784222 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.784123    2577 generic.go:356] "Generic (PLEG): container finished" podID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" containerID="42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413" exitCode=0
Sep 10 14:05:51.784222 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.784171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"
Sep 10 14:05:51.784222 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.784183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" event={"ID":"3f49ac24-0adf-4262-bfbd-d9d3dff9f403","Type":"ContainerDied","Data":"42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413"}
Sep 10 14:05:51.784507 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.784251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f" event={"ID":"3f49ac24-0adf-4262-bfbd-d9d3dff9f403","Type":"ContainerDied","Data":"deb87e6798ee36ae00497dea44705772fbc79beb366c313c816050cc4dc981f6"}
Sep 10 14:05:51.784507 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.784284    2577 scope.go:117] "RemoveContainer" containerID="42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413"
Sep 10 14:05:51.796758 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.796734    2577 scope.go:117] "RemoveContainer" containerID="42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413"
Sep 10 14:05:51.797163 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:05:51.797135    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413\": container with ID starting with 42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413 not found: ID does not exist" containerID="42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413"
Sep 10 14:05:51.797278 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.797170    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413"} err="failed to get container status \"42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413\": rpc error: code = NotFound desc = could not find container \"42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413\": container with ID starting with 42922b7daf9d3ad19a5d48e036aa236b114476b4214e9e7bb9704206a2019413 not found: ID does not exist"
Sep 10 14:05:51.889866 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.889835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk4mt\" (UniqueName: \"kubernetes.io/projected/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-kube-api-access-gk4mt\") pod \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") "
Sep 10 14:05:51.890038 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.889912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-extensions-socket-volume\") pod \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\" (UID: \"3f49ac24-0adf-4262-bfbd-d9d3dff9f403\") "
Sep 10 14:05:51.890297 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.890270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f49ac24-0adf-4262-bfbd-d9d3dff9f403" (UID: "3f49ac24-0adf-4262-bfbd-d9d3dff9f403"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:05:51.891907 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.891886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-kube-api-access-gk4mt" (OuterVolumeSpecName: "kube-api-access-gk4mt") pod "3f49ac24-0adf-4262-bfbd-d9d3dff9f403" (UID: "3f49ac24-0adf-4262-bfbd-d9d3dff9f403"). InnerVolumeSpecName "kube-api-access-gk4mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:05:51.990742 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.990693    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4mt\" (UniqueName: \"kubernetes.io/projected/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-kube-api-access-gk4mt\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:05:51.990742 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:51.990731    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f49ac24-0adf-4262-bfbd-d9d3dff9f403-extensions-socket-volume\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:05:52.120252 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:52.120219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"]
Sep 10 14:05:52.131636 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:52.131597    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rpj8f"]
Sep 10 14:05:52.452310 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:52.452225    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" path="/var/lib/kubelet/pods/3f49ac24-0adf-4262-bfbd-d9d3dff9f403/volumes"
Sep 10 14:05:56.643032 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:56.643006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 14:05:56.644883 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:05:56.644853    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xjdx_9fe748bb-3e14-48ac-af5e-bf8fafecf1ab/ovn-acl-logging/0.log"
Sep 10 14:06:52.395194 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.395099    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"]
Sep 10 14:06:52.395631 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.395568    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" containerName="manager"
Sep 10 14:06:52.395631 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.395594    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" containerName="manager"
Sep 10 14:06:52.395708 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.395652    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f49ac24-0adf-4262-bfbd-d9d3dff9f403" containerName="manager"
Sep 10 14:06:52.398966 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.398944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.402324 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.402304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sztmq\""
Sep 10 14:06:52.424747 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.424709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"]
Sep 10 14:06:52.499460 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.499416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9pb\" (UniqueName: \"kubernetes.io/projected/aeaf8a6a-d231-4760-9f62-c25756c3ae97-kube-api-access-nf9pb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nwlcz\" (UID: \"aeaf8a6a-d231-4760-9f62-c25756c3ae97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.499722 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.499541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeaf8a6a-d231-4760-9f62-c25756c3ae97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nwlcz\" (UID: \"aeaf8a6a-d231-4760-9f62-c25756c3ae97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.600696 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.600651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeaf8a6a-d231-4760-9f62-c25756c3ae97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nwlcz\" (UID: \"aeaf8a6a-d231-4760-9f62-c25756c3ae97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.600890 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.600756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9pb\" (UniqueName: \"kubernetes.io/projected/aeaf8a6a-d231-4760-9f62-c25756c3ae97-kube-api-access-nf9pb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nwlcz\" (UID: \"aeaf8a6a-d231-4760-9f62-c25756c3ae97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.601057 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.601035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeaf8a6a-d231-4760-9f62-c25756c3ae97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nwlcz\" (UID: \"aeaf8a6a-d231-4760-9f62-c25756c3ae97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.614793 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.614758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9pb\" (UniqueName: \"kubernetes.io/projected/aeaf8a6a-d231-4760-9f62-c25756c3ae97-kube-api-access-nf9pb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nwlcz\" (UID: \"aeaf8a6a-d231-4760-9f62-c25756c3ae97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.709565 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.709450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:52.994934 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:52.994908    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"]
Sep 10 14:06:52.996682 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:06:52.996650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaf8a6a_d231_4760_9f62_c25756c3ae97.slice/crio-a29a70d080db7e81dd8aee4b03321b387ae943233f3e5986b2f748dffb74b8c9 WatchSource:0}: Error finding container a29a70d080db7e81dd8aee4b03321b387ae943233f3e5986b2f748dffb74b8c9: Status 404 returned error can't find the container with id a29a70d080db7e81dd8aee4b03321b387ae943233f3e5986b2f748dffb74b8c9
Sep 10 14:06:53.112161 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:53.112122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz" event={"ID":"aeaf8a6a-d231-4760-9f62-c25756c3ae97","Type":"ContainerStarted","Data":"873660d25339d1cc9a4b2056a5b5c48dc74821c8da5c2db54f07dfd72b41c0a1"}
Sep 10 14:06:53.112340 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:53.112181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz" event={"ID":"aeaf8a6a-d231-4760-9f62-c25756c3ae97","Type":"ContainerStarted","Data":"a29a70d080db7e81dd8aee4b03321b387ae943233f3e5986b2f748dffb74b8c9"}
Sep 10 14:06:53.112340 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:53.112229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:06:53.160883 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:53.160823    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz" podStartSLOduration=1.160803758 podStartE2EDuration="1.160803758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:06:53.158996476 +0000 UTC m=+2157.183811179" watchObservedRunningTime="2026-09-10 14:06:53.160803758 +0000 UTC m=+2157.185618461"
Sep 10 14:06:56.834235 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:56.834201    2577 scope.go:117] "RemoveContainer" containerID="f0bbdd18697f0fea4c915d067ad676f5e61c9746682a227520ae94ff6d2e92a0"
Sep 10 14:06:56.842779 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:06:56.842756    2577 scope.go:117] "RemoveContainer" containerID="50a35ff91ed6d60102193b14307c67cc45dd9d794f8c703e3a0c63defdcc124e"
Sep 10 14:07:04.120461 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:07:04.120424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nwlcz"
Sep 10 14:09:01.168837 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.168798    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"]
Sep 10 14:09:01.172303 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.172283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.189393 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.189368    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"]
Sep 10 14:09:01.330041 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.330007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9900496c-9901-460f-8ed9-5bc63f9a8c9a-maas-api-tls\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.330228 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.330080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9900496c-9901-460f-8ed9-5bc63f9a8c9a-openshift-service-ca\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.330228 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.330118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n289f\" (UniqueName: \"kubernetes.io/projected/9900496c-9901-460f-8ed9-5bc63f9a8c9a-kube-api-access-n289f\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.330228 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.330137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9900496c-9901-460f-8ed9-5bc63f9a8c9a-metrics-certs\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.430733 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.430634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9900496c-9901-460f-8ed9-5bc63f9a8c9a-maas-api-tls\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.430909 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.430742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9900496c-9901-460f-8ed9-5bc63f9a8c9a-openshift-service-ca\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.430909 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.430821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n289f\" (UniqueName: \"kubernetes.io/projected/9900496c-9901-460f-8ed9-5bc63f9a8c9a-kube-api-access-n289f\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.430909 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.430854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9900496c-9901-460f-8ed9-5bc63f9a8c9a-metrics-certs\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.431555 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.431532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9900496c-9901-460f-8ed9-5bc63f9a8c9a-openshift-service-ca\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.433208 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.433188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9900496c-9901-460f-8ed9-5bc63f9a8c9a-maas-api-tls\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.433287 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.433227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9900496c-9901-460f-8ed9-5bc63f9a8c9a-metrics-certs\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.448123 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.448101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n289f\" (UniqueName: \"kubernetes.io/projected/9900496c-9901-460f-8ed9-5bc63f9a8c9a-kube-api-access-n289f\") pod \"maas-api-55d65dcfc4-btlsk\" (UID: \"9900496c-9901-460f-8ed9-5bc63f9a8c9a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.482555 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.482530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:01.775709 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.775681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"]
Sep 10 14:09:01.778012 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:09:01.777986    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9900496c_9901_460f_8ed9_5bc63f9a8c9a.slice/crio-fa8af1a58bfeafb0085cfa0a2879062e2d731d1da1166dcd9a3c229a01cac689 WatchSource:0}: Error finding container fa8af1a58bfeafb0085cfa0a2879062e2d731d1da1166dcd9a3c229a01cac689: Status 404 returned error can't find the container with id fa8af1a58bfeafb0085cfa0a2879062e2d731d1da1166dcd9a3c229a01cac689
Sep 10 14:09:01.779265 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.779245    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:09:01.786560 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:01.786535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk" event={"ID":"9900496c-9901-460f-8ed9-5bc63f9a8c9a","Type":"ContainerStarted","Data":"fa8af1a58bfeafb0085cfa0a2879062e2d731d1da1166dcd9a3c229a01cac689"}
Sep 10 14:09:02.793318 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:02.793238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk" event={"ID":"9900496c-9901-460f-8ed9-5bc63f9a8c9a","Type":"ContainerStarted","Data":"d88b7d523fb8079e8b3a365f31688de7bb372a0c549612cfee5994167d7d6a08"}
Sep 10 14:09:02.793684 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:02.793348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:02.825677 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:02.825625    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk" podStartSLOduration=1.163488401 podStartE2EDuration="1.825609767s" totalImagesPullingTime="662.121366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:09:02.823493195 +0000 UTC m=+2286.848307899" watchObservedRunningTime="2026-09-10 14:09:02.825609767 +0000 UTC m=+2286.850424469"
Sep 10 14:09:08.804423 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:08.804388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-btlsk"
Sep 10 14:09:08.887922 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:08.887893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"]
Sep 10 14:09:08.888133 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:08.888108    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb" podUID="28c69857-115b-4bfd-9b2c-9a8c8199d397" containerName="maas-api" containerID="cri-o://edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba" gracePeriod=30
Sep 10 14:09:10.141527 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.141504    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 14:09:10.314493 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.314411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") pod \"28c69857-115b-4bfd-9b2c-9a8c8199d397\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") "
Sep 10 14:09:10.314493 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.314484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c69857-115b-4bfd-9b2c-9a8c8199d397-openshift-service-ca\") pod \"28c69857-115b-4bfd-9b2c-9a8c8199d397\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") "
Sep 10 14:09:10.314724 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.314546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khzkf\" (UniqueName: \"kubernetes.io/projected/28c69857-115b-4bfd-9b2c-9a8c8199d397-kube-api-access-khzkf\") pod \"28c69857-115b-4bfd-9b2c-9a8c8199d397\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") "
Sep 10 14:09:10.314724 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.314634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-maas-api-tls\") pod \"28c69857-115b-4bfd-9b2c-9a8c8199d397\" (UID: \"28c69857-115b-4bfd-9b2c-9a8c8199d397\") "
Sep 10 14:09:10.314942 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.314918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c69857-115b-4bfd-9b2c-9a8c8199d397-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28c69857-115b-4bfd-9b2c-9a8c8199d397" (UID: "28c69857-115b-4bfd-9b2c-9a8c8199d397"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:09:10.315065 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.315044    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c69857-115b-4bfd-9b2c-9a8c8199d397-openshift-service-ca\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:09:10.316798 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.316773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28c69857-115b-4bfd-9b2c-9a8c8199d397" (UID: "28c69857-115b-4bfd-9b2c-9a8c8199d397"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:09:10.316898 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.316853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c69857-115b-4bfd-9b2c-9a8c8199d397-kube-api-access-khzkf" (OuterVolumeSpecName: "kube-api-access-khzkf") pod "28c69857-115b-4bfd-9b2c-9a8c8199d397" (UID: "28c69857-115b-4bfd-9b2c-9a8c8199d397"). InnerVolumeSpecName "kube-api-access-khzkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:09:10.316956 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.316901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28c69857-115b-4bfd-9b2c-9a8c8199d397" (UID: "28c69857-115b-4bfd-9b2c-9a8c8199d397"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:09:10.415844 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.415812    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-metrics-certs\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:09:10.415844 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.415840    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khzkf\" (UniqueName: \"kubernetes.io/projected/28c69857-115b-4bfd-9b2c-9a8c8199d397-kube-api-access-khzkf\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:09:10.415844 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.415851    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c69857-115b-4bfd-9b2c-9a8c8199d397-maas-api-tls\") on node \"ip-10-0-130-202.ec2.internal\" DevicePath \"\""
Sep 10 14:09:10.835935 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.835904    2577 generic.go:356] "Generic (PLEG): container finished" podID="28c69857-115b-4bfd-9b2c-9a8c8199d397" containerID="edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba" exitCode=0
Sep 10 14:09:10.836105 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.835953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb" event={"ID":"28c69857-115b-4bfd-9b2c-9a8c8199d397","Type":"ContainerDied","Data":"edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba"}
Sep 10 14:09:10.836105 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.835962    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"
Sep 10 14:09:10.836105 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.835994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb" event={"ID":"28c69857-115b-4bfd-9b2c-9a8c8199d397","Type":"ContainerDied","Data":"d47bb3f56decb84f8bb426de6774b2831bc02821f0d5965584ba3634eea93f65"}
Sep 10 14:09:10.836105 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.836015    2577 scope.go:117] "RemoveContainer" containerID="edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba"
Sep 10 14:09:10.844841 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.844824    2577 scope.go:117] "RemoveContainer" containerID="edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba"
Sep 10 14:09:10.845109 ip-10-0-130-202 kubenswrapper[2577]: E0910 14:09:10.845090    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba\": container with ID starting with edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba not found: ID does not exist" containerID="edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba"
Sep 10 14:09:10.845153 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.845121    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba"} err="failed to get container status \"edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba\": rpc error: code = NotFound desc = could not find container \"edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba\": container with ID starting with edc486d4c59c9a4a8411aa90b52fe65a07fdb03e40b3bd0b1517f91a1d5969ba not found: ID does not exist"
Sep 10 14:09:10.879281 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.879260    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"]
Sep 10 14:09:10.888310 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:10.888289    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-nxltb"]
Sep 10 14:09:12.452864 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:12.452828    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c69857-115b-4bfd-9b2c-9a8c8199d397" path="/var/lib/kubelet/pods/28c69857-115b-4bfd-9b2c-9a8c8199d397/volumes"
Sep 10 14:09:28.710612 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:28.710571    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xbfl8_92516cc8-609a-4371-9a79-3543d5f5bbde/manager/0.log"
Sep 10 14:09:28.840337 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:28.840308    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l5frs_833dcd84-968d-4ffb-8b5d-7f86109bcaf8/manager/0.log"
Sep 10 14:09:28.968469 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:28.968393    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9955587-6r22m_5b6b22ad-31e9-4a3e-8648-7adcf17e1b10/manager/0.log"
Sep 10 14:09:29.094054 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:29.094028    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mgnch_9447ffbd-881b-4b7b-9380-7abcd2c3bdfb/server/0.log"
Sep 10 14:09:29.615349 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:29.615321    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-hhc9q_65d0d623-a0b6-4041-9202-a2037a64c1f1/copy-manifests/0.log"
Sep 10 14:09:29.623828 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:29.623803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-hhc9q_65d0d623-a0b6-4041-9202-a2037a64c1f1/manager/0.log"
Sep 10 14:09:29.969254 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:29.969183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55d65dcfc4-btlsk_9900496c-9901-460f-8ed9-5bc63f9a8c9a/maas-api/0.log"
Sep 10 14:09:31.767454 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:31.767418    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-nwlcz_aeaf8a6a-d231-4760-9f62-c25756c3ae97/manager/0.log"
Sep 10 14:09:32.014129 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:32.014098    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mwq8p_449c08d0-c09c-4c86-9e43-926d524709cc/manager/0.log"
Sep 10 14:09:32.361515 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:32.361482    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k_d20f11b0-7ada-4bd5-a6b5-e93d2b58994b/istio-proxy/0.log"
Sep 10 14:09:33.728641 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:33.728606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9d67v_fa0b1a0c-e74b-4328-98cc-a15d1aa40c62/main/0.log"
Sep 10 14:09:34.099452 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:34.099417    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flgx5m_9b50b7f8-fd3a-45cd-8306-22d62e48c864/main/0.log"
Sep 10 14:09:34.234775 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:34.234728    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jjx72_9df2ceb3-9df0-4dda-a406-a2ecaf1d2bf0/main/0.log"
Sep 10 14:09:43.332902 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:43.332865    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-695ls_a404e583-94f1-455b-b237-16adac92fe09/global-pull-secret-syncer/0.log"
Sep 10 14:09:43.570219 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:43.570192    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wsz7q_b49da1e9-edb6-47e7-a9f0-25bc1d3f7112/konnectivity-agent/0.log"
Sep 10 14:09:43.664946 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:43.664863    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-202.ec2.internal_bb5a6d65385ac6133bd7865d2c10033e/haproxy/0.log"
Sep 10 14:09:49.131052 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:49.131021    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-nwlcz_aeaf8a6a-d231-4760-9f62-c25756c3ae97/manager/0.log"
Sep 10 14:09:49.335182 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:49.335145    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mwq8p_449c08d0-c09c-4c86-9e43-926d524709cc/manager/0.log"
Sep 10 14:09:50.964022 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:50.963941    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/alertmanager/0.log"
Sep 10 14:09:50.990326 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:50.990299    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/config-reloader/0.log"
Sep 10 14:09:51.019457 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.019432    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/kube-rbac-proxy-web/0.log"
Sep 10 14:09:51.045386 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.045360    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/kube-rbac-proxy/0.log"
Sep 10 14:09:51.074770 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.074746    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/kube-rbac-proxy-metric/0.log"
Sep 10 14:09:51.106647 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.106621    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/prom-label-proxy/0.log"
Sep 10 14:09:51.135884 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.135860    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca4f0559-e610-4b7a-ad16-2b7a4df308bf/init-config-reloader/0.log"
Sep 10 14:09:51.234560 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.234489    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-2snc9_0d089c83-40a2-4b68-b18f-fd20f010483d/kube-state-metrics/0.log"
Sep 10 14:09:51.274983 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.274955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-2snc9_0d089c83-40a2-4b68-b18f-fd20f010483d/kube-rbac-proxy-main/0.log"
Sep 10 14:09:51.301538 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.301505    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-2snc9_0d089c83-40a2-4b68-b18f-fd20f010483d/kube-rbac-proxy-self/0.log"
Sep 10 14:09:51.338364 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.338343    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fc48648d6-rpkvd_2b521937-f3ce-4996-94db-e978d886b5d3/metrics-server/0.log"
Sep 10 14:09:51.370052 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.370010    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-zrpv9_ef1f53f6-423a-434f-aa64-ef3f2d4e96f2/monitoring-plugin/0.log"
Sep 10 14:09:51.516946 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.516877    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c98nt_1be5bf26-1e6a-40be-a453-990bfb64e7c7/node-exporter/0.log"
Sep 10 14:09:51.546738 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.546712    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c98nt_1be5bf26-1e6a-40be-a453-990bfb64e7c7/kube-rbac-proxy/0.log"
Sep 10 14:09:51.582627 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.582605    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c98nt_1be5bf26-1e6a-40be-a453-990bfb64e7c7/init-textfile/0.log"
Sep 10 14:09:51.716711 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.716684    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gvp4m_1379c126-248e-4ca8-ae92-091740b115b2/kube-rbac-proxy-main/0.log"
Sep 10 14:09:51.804851 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.804779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gvp4m_1379c126-248e-4ca8-ae92-091740b115b2/kube-rbac-proxy-self/0.log"
Sep 10 14:09:51.854104 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.854083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gvp4m_1379c126-248e-4ca8-ae92-091740b115b2/openshift-state-metrics/0.log"
Sep 10 14:09:51.908891 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.908867    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/prometheus/0.log"
Sep 10 14:09:51.951800 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.951767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/config-reloader/0.log"
Sep 10 14:09:51.989271 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:51.989245    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/thanos-sidecar/0.log"
Sep 10 14:09:52.021990 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.021969    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/kube-rbac-proxy-web/0.log"
Sep 10 14:09:52.047993 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.047970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/kube-rbac-proxy/0.log"
Sep 10 14:09:52.087005 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.086978    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/kube-rbac-proxy-thanos/0.log"
Sep 10 14:09:52.116074 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.116051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1857d2fc-d81f-4bac-a30d-a839e3974be7/init-config-reloader/0.log"
Sep 10 14:09:52.158952 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.158926    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-5dd89_a3758b02-ff8d-4023-8e56-10f9d5e25be0/prometheus-operator/0.log"
Sep 10 14:09:52.184932 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.184911    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-5dd89_a3758b02-ff8d-4023-8e56-10f9d5e25be0/kube-rbac-proxy/0.log"
Sep 10 14:09:52.256666 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.256642    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-95d898cb-kb8mp_20044fa4-cf73-436a-ad78-a762e91f3d05/telemeter-client/0.log"
Sep 10 14:09:52.284704 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.284682    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-95d898cb-kb8mp_20044fa4-cf73-436a-ad78-a762e91f3d05/reload/0.log"
Sep 10 14:09:52.311801 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.311778    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-95d898cb-kb8mp_20044fa4-cf73-436a-ad78-a762e91f3d05/kube-rbac-proxy/0.log"
Sep 10 14:09:52.344214 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.344135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b7665cccd-249vg_308713cb-bd60-4b14-9b27-5c4e700091da/thanos-query/0.log"
Sep 10 14:09:52.388750 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.388716    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b7665cccd-249vg_308713cb-bd60-4b14-9b27-5c4e700091da/kube-rbac-proxy-web/0.log"
Sep 10 14:09:52.414381 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.414350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b7665cccd-249vg_308713cb-bd60-4b14-9b27-5c4e700091da/kube-rbac-proxy/0.log"
Sep 10 14:09:52.454919 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.454887    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b7665cccd-249vg_308713cb-bd60-4b14-9b27-5c4e700091da/prom-label-proxy/0.log"
Sep 10 14:09:52.488338 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.488308    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b7665cccd-249vg_308713cb-bd60-4b14-9b27-5c4e700091da/kube-rbac-proxy-rules/0.log"
Sep 10 14:09:52.514367 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:52.514343    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b7665cccd-249vg_308713cb-bd60-4b14-9b27-5c4e700091da/kube-rbac-proxy-metrics/0.log"
Sep 10 14:09:54.668151 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.668118    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"]
Sep 10 14:09:54.668601 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.668570    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c69857-115b-4bfd-9b2c-9a8c8199d397" containerName="maas-api"
Sep 10 14:09:54.668601 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.668598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c69857-115b-4bfd-9b2c-9a8c8199d397" containerName="maas-api"
Sep 10 14:09:54.668719 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.668706    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c69857-115b-4bfd-9b2c-9a8c8199d397" containerName="maas-api"
Sep 10 14:09:54.672224 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.672209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.674690 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.674666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mbn6s\"/\"kube-root-ca.crt\""
Sep 10 14:09:54.674824 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.674704    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mbn6s\"/\"openshift-service-ca.crt\""
Sep 10 14:09:54.675363 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.675345    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mbn6s\"/\"default-dockercfg-2lwpv\""
Sep 10 14:09:54.690457 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.690435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"]
Sep 10 14:09:54.722708 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.722681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-proc\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.722831 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.722724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctl95\" (UniqueName: \"kubernetes.io/projected/d68e0467-b06f-4f55-95a7-30b80bdbf320-kube-api-access-ctl95\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.722831 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.722742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-sys\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.722906 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.722848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-podres\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.722906 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.722886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-lib-modules\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.823781 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-proc\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.823932 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctl95\" (UniqueName: \"kubernetes.io/projected/d68e0467-b06f-4f55-95a7-30b80bdbf320-kube-api-access-ctl95\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.823932 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-sys\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.823932 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-podres\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.823932 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-proc\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.823932 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-lib-modules\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.824130 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.823952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-sys\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.824130 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.824007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-podres\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.824130 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.824015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d68e0467-b06f-4f55-95a7-30b80bdbf320-lib-modules\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.833694 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.833663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctl95\" (UniqueName: \"kubernetes.io/projected/d68e0467-b06f-4f55-95a7-30b80bdbf320-kube-api-access-ctl95\") pod \"perf-node-gather-daemonset-rp6dd\" (UID: \"d68e0467-b06f-4f55-95a7-30b80bdbf320\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:54.982325 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:54.982228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:55.258850 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:55.258820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"]
Sep 10 14:09:55.260786 ip-10-0-130-202 kubenswrapper[2577]: W0910 14:09:55.260759    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd68e0467_b06f_4f55_95a7_30b80bdbf320.slice/crio-bfe9d96c19f8b335ff5408b8906f0b2e6d758e8ba3a7c1160592d1cbd8506008 WatchSource:0}: Error finding container bfe9d96c19f8b335ff5408b8906f0b2e6d758e8ba3a7c1160592d1cbd8506008: Status 404 returned error can't find the container with id bfe9d96c19f8b335ff5408b8906f0b2e6d758e8ba3a7c1160592d1cbd8506008
Sep 10 14:09:56.070918 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:56.070875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd" event={"ID":"d68e0467-b06f-4f55-95a7-30b80bdbf320","Type":"ContainerStarted","Data":"9d3a039e3192bfa72fc03a269dd2d40d5960157be448e94718bd58358960644e"}
Sep 10 14:09:56.071341 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:56.070935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd" event={"ID":"d68e0467-b06f-4f55-95a7-30b80bdbf320","Type":"ContainerStarted","Data":"bfe9d96c19f8b335ff5408b8906f0b2e6d758e8ba3a7c1160592d1cbd8506008"}
Sep 10 14:09:56.071341 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:56.070956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:09:56.095803 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:56.095754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd" podStartSLOduration=2.09574066 podStartE2EDuration="2.09574066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:09:56.093197299 +0000 UTC m=+2340.118012001" watchObservedRunningTime="2026-09-10 14:09:56.09574066 +0000 UTC m=+2340.120555362"
Sep 10 14:09:57.071074 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:57.071042    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2xt2_fb53d9a8-99c6-4a02-963e-0a5efa640280/dns/0.log"
Sep 10 14:09:57.132013 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:57.131985    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2xt2_fb53d9a8-99c6-4a02-963e-0a5efa640280/kube-rbac-proxy/0.log"
Sep 10 14:09:57.193242 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:57.193163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmmtd_fb7c86e2-0c7a-417b-9b6f-ea9a33f95795/dns-node-resolver/0.log"
Sep 10 14:09:57.922953 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:57.922921    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bhlmv_a4780b69-677c-48f7-b972-e3a66b6025df/node-ca/0.log"
Sep 10 14:09:59.129634 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:09:59.129600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k_d20f11b0-7ada-4bd5-a6b5-e93d2b58994b/istio-proxy/0.log"
Sep 10 14:10:00.652979 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:00.652948    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d7jkb_a8e27269-7607-4483-b8c3-b161b07da691/serve-healthcheck-canary/0.log"
Sep 10 14:10:01.466236 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:01.466204    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zp5t_68a8e641-6318-47ab-9605-39cc5dfb3c81/kube-rbac-proxy/0.log"
Sep 10 14:10:01.508463 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:01.508433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zp5t_68a8e641-6318-47ab-9605-39cc5dfb3c81/exporter/0.log"
Sep 10 14:10:01.550504 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:01.550473    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zp5t_68a8e641-6318-47ab-9605-39cc5dfb3c81/extractor/0.log"
Sep 10 14:10:02.085621 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:02.085591    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-rp6dd"
Sep 10 14:10:03.865784 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:03.865755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k_d20f11b0-7ada-4bd5-a6b5-e93d2b58994b/istio-proxy/0.log"
Sep 10 14:10:04.478058 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:04.478033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xbfl8_92516cc8-609a-4371-9a79-3543d5f5bbde/manager/0.log"
Sep 10 14:10:04.752197 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:04.752166    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l5frs_833dcd84-968d-4ffb-8b5d-7f86109bcaf8/manager/0.log"
Sep 10 14:10:04.823732 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:04.823706    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9955587-6r22m_5b6b22ad-31e9-4a3e-8648-7adcf17e1b10/manager/0.log"
Sep 10 14:10:04.852287 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:04.852262    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mgnch_9447ffbd-881b-4b7b-9380-7abcd2c3bdfb/server/0.log"
Sep 10 14:10:05.285294 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:05.285264    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-hhc9q_65d0d623-a0b6-4041-9202-a2037a64c1f1/manager/0.log"
Sep 10 14:10:05.316170 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:05.316144    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-hhc9q_65d0d623-a0b6-4041-9202-a2037a64c1f1/copy-manifests/0.log"
Sep 10 14:10:06.129341 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:06.129313    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cxnh8k_d20f11b0-7ada-4bd5-a6b5-e93d2b58994b/istio-proxy/0.log"
Sep 10 14:10:06.681627 ip-10-0-130-202 kubenswrapper[2577]: I0910 14:10:06.681595    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59689cfc58-79cns_ed44035f-cb33-44c2-9419-db0ed9fafadd/manager/0.log"