Sep 07 16:17:51.303856 ip-10-0-129-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 07 16:17:51.303869 ip-10-0-129-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 07 16:17:51.303879 ip-10-0-129-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 07 16:17:51.304180 ip-10-0-129-87 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 07 16:18:01.422212 ip-10-0-129-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 07 16:18:01.422232 ip-10-0-129-87 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 10fa6d49212141369f6438039cb53bf2 --
Sep 07 16:19:53.717086 ip-10-0-129-87 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 16:19:54.076365 ip-10-0-129-87 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 16:19:54.076365 ip-10-0-129-87 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 16:19:54.076365 ip-10-0-129-87 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 16:19:54.076365 ip-10-0-129-87 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 16:19:54.076365 ip-10-0-129-87 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 16:19:54.078797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.078668    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 16:19:54.082379 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082357    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 16:19:54.082379 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082377    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 16:19:54.082379 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082381    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082385    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082388    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082391    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082394    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082397    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082401    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082405    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082408    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082411    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082416    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082419    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082422    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082425    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082427    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082430    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082433    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082435    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082438    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 16:19:54.082476 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082441    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082444    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082448    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082451    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082454    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082457    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082460    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082462    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082465    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082467    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082470    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082472    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082475    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082477    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082481    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082484    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082487    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082489    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082492    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 16:19:54.082926 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082494    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082496    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082499    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082502    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082506    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082509    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082511    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082514    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082516    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082518    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082521    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082523    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082525    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082528    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082530    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082534    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082536    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082539    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082541    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 16:19:54.083376 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082544    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082546    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082549    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082551    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082554    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082556    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082559    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082561    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082564    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082566    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082569    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082571    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082573    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082576    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082579    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082581    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082583    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082586    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082588    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082591    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 16:19:54.083881 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082594    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082597    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082600    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082602    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082604    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082607    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082609    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082612    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082615    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082617    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082620    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082622    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082624    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082627    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082630    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082632    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082635    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082637    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082639    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.082654    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 16:19:54.084347 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083358    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083365    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083368    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083370    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083373    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083376    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083378    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083381    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083383    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083386    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083388    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083391    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083396    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083399    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083402    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083405    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083408    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083412    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083416    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 16:19:54.084847 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083419    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083422    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083426    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083429    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083432    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083435    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083437    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083440    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083443    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083446    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083448    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083451    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083453    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083456    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083458    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083461    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083463    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083467    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083470    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083473    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 16:19:54.085315 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083476    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083479    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083481    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083483    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083486    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083489    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083492    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083495    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083497    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083499    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083502    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083505    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083507    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083510    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083512    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083516    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083519    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083522    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083525    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 16:19:54.085812 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083528    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083530    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083533    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083535    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083537    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083540    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083542    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083544    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083547    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083550    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083552    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083555    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083557    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083561    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083563    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083565    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083568    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083570    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083572    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083575    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 16:19:54.086284 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083578    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083580    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083583    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083585    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083587    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083590    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083592    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083595    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083597    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083600    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083602    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083604    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083607    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083609    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083612    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083614    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083617    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083619    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083621    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083624    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 16:19:54.086779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.083626    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084038    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084043    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084046    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084049    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084051    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084054    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084057    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084060    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084062    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084065    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084069    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084074    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084077    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084079    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084082    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084084    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084087    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084089    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084092    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 16:19:54.087270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084094    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084097    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084099    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084102    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084104    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084107    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084109    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084111    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084114    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084116    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084118    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084121    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084124    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084126    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084129    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084132    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084134    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084136    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084138    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 16:19:54.087755 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084141    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084143    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084145    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084151    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084154    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084157    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084162    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084165    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084167    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084170    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084173    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084175    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084177    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084180    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084182    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084184    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084187    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084189    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084192    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 16:19:54.088197 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084194    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084196    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084199    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084201    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084204    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084206    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084208    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084211    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084214    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084216    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084218    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084221    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084223    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084225    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084227    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084232    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084235    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084238    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084241    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084243    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 16:19:54.088696 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084246    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084249    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084251    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084253    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084256    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084258    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084260    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084263    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084265    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084267    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084271    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084274    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084277    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084280    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084282    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084284    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084287    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084289    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084292    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084295    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 16:19:54.089164 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084298    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.084301    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084382    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084416    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084422    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084427    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084431    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084435    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084441    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084445    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084448    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084451    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084454    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084458    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084461    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084464    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084467    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084470    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084472    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084475    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084478    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084481    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084484    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084487    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 16:19:54.089685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084490    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084494    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084497    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084500    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084503    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084507    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084510    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084513    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084516    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084519    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084523    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084526    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084528    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084532    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084535    2577 flags.go:64] FLAG: --enable-server="true"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084539    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084545    2577 flags.go:64] FLAG: --event-burst="100"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084547    2577 flags.go:64] FLAG: --event-qps="50"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084550    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084553    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084556    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084560    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084566    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084569    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084572    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 07 16:19:54.090248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084575    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084578    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084580    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084583    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084586    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084589    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084591    2577 flags.go:64] FLAG: --feature-gates=""
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084595    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084598    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084601    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084604    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084607    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084610    2577 flags.go:64] FLAG: --help="false"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084613    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084616    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084618    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084621    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084624    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084628    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084630    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084633    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084636    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084640    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 16:19:54.090845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084661    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084665    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084667    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084670    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084673    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084676    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084679    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084682    2577 flags.go:64] FLAG: --lock-file=""
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084684    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084687    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084690    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084696    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084698    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084701    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084703    2577 flags.go:64] FLAG: --logging-format="text"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084706    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084709    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084712    2577 flags.go:64] FLAG: --manifest-url=""
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084715    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084719    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084723    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084726    2577 flags.go:64] FLAG: --max-pods="110"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084729    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084732    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084734    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 16:19:54.091426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084737    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084740    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084743    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084746    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084753    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084756    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084761    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084766    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084769    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084775    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084778    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084780    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084783    2577 flags.go:64] FLAG: --port="10250"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084786    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084789    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0491b2bf28b0ebc21"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084792    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084795    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084798    2577 flags.go:64] FLAG: --register-node="true"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084801    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084804    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084807    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084810    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084812    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084816    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084819    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 16:19:54.092097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084821    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084824    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084827    2577 flags.go:64] FLAG: --runonce="false"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084829    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084832    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084835    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084838    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084840    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084843    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084846    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084849    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084852    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084855    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084859    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084862    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084865    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084868    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084871    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084876    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084879    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084882    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084885    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084888    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084891    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084893    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 16:19:54.092971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084896    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084899    2577 flags.go:64] FLAG: --v="2"
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084907    2577 flags.go:64] FLAG: --version="false"
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084911    2577 flags.go:64] FLAG: --vmodule=""
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084915    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.084918    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085032    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085035    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085038    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085041    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085043    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085046    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085048    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085052    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085056    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085059    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085061    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085064    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085067    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085069    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085073    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 16:19:54.093872 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085076    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085079    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085082    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085084    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085087    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085089    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085091    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085094    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085096    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085098    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085101    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085104    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085106    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085108    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085111    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085113    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085115    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085118    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085120    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085122    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 16:19:54.094389 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085124    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085127    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085129    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085132    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085134    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085137    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085140    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085142    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085144    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085147    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085149    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085153    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085155    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085158    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085160    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085162    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085165    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085167    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085169    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 16:19:54.094880 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085172    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085174    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085176    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085179    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085181    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085183    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085186    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085188    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085190    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085193    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085195    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085197    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085199    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085202    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085204    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085206    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085208    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085211    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085213    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085215    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 16:19:54.095342 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085218    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085220    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085222    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085225    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085228    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085231    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085233    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085235    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085237    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085240    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085242    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085244    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085247    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085250    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085253    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085256    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085258    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085261    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085263    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085266    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 16:19:54.095949 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085268    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085270    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085273    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085275    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.085277    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.086001    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.094847    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.094993    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.095733    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.095741    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095802    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095808    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095811    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095814    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 16:19:54.096437 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095817    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095820    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095823    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095825    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095828    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095830    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095833    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095835    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095838    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095840    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095842    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095846    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095848    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095851    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095854    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095856    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095859    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095863    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095866    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 16:19:54.096795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095869    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095872    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095875    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095878    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095880    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095883    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095886    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095889    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095892    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095895    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095899    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095901    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095905    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095907    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095910    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095912    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095915    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095917    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095920    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095922    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 16:19:54.097233 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095925    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095927    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095930    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095933    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095936    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095938    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095941    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095944    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095946    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095950    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095954    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095957    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095960    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095963    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095966    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095969    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095971    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095975    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095979    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 16:19:54.097796 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095982    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095985    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095988    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095991    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095993    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095996    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.095999    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096001    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096004    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096006    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096009    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096011    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096013    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096016    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096018    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096021    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096024    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096027    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096030    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096032    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 16:19:54.098239 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096035    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096037    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096040    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096042    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096045    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096047    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096049    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096052    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096054    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096057    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096059    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096062    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096066    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096068    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096071    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096073    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 16:19:54.098714 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096075    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.096080    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096204    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096209    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096213    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096215    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096218    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096220    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096223    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096225    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096227    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096230    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096233    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096235    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096239    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 16:19:54.099138 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096242    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096244    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096247    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096249    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096252    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096254    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096257    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096260    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096262    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096265    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096267    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096269    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096272    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096275    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096277    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096280    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096283    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096285    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096287    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096290    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 16:19:54.099511 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096292    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096295    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096297    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096299    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096302    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096305    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096307    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096309    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096312    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096315    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096317    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096319    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096323    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096325    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096327    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096330    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096332    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096335    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096338    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 16:19:54.100061 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096341    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096343    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096346    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096348    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096350    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096354    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096357    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096360    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096364    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096366    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096369    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096372    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096375    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096377    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096380    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096382    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096384    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096387    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096389    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 16:19:54.100559 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096392    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096394    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096397    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096399    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096401    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096404    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096406    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096409    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096411    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096413    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096416    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096418    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096420    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096423    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096425    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096427    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096429    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096432    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096434    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096438    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 16:19:54.101045 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096440    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096443    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096446    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096448    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096451    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096453    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096456    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:54.096458    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.096463    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.096589    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.098699    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.099477    2577 server.go:1019] "Starting client certificate rotation"
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.099600    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 16:19:54.101850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.100469    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 16:19:54.102347 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.102080    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 16:19:54.103294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.103270    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 16:19:54.114939 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.114922    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 07 16:19:54.117980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.117960    2577 log.go:25] "Validated CRI v1 image API"
Sep 07 16:19:54.118845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.118828    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 16:19:54.124457 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.124435    2577 fs.go:135] Filesystem UUIDs: map[507ab67b-ec4a-447c-9882-08e1408341f8:/dev/nvme0n1p4 75a72703-7856-4a9f-abb9-07d76056d805:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 07 16:19:54.124552 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.124454    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 16:19:54.126502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.126481    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 16:19:54.131870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.131764    2577 manager.go:216] Machine: {Timestamp:2026-09-07 16:19:54.130551523 +0000 UTC m=+0.305194472 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3194266 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e54e4a6fa5642e5a4249fc32ebb23 SystemUUID:ec2e54e4-a6fa-5642-e5a4-249fc32ebb23 BootID:10fa6d49-2121-4136-9f64-38039cb53bf2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:16:79:e3:27:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:16:79:e3:27:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:34:55:f2:95:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 16:19:54.131870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.131862    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 16:19:54.132001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.131980    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 16:19:54.136034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.136000    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 16:19:54.136186 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.136037    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 16:19:54.137224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.137212    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 16:19:54.137272 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.137227    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 16:19:54.137272 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.137235    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 16:19:54.137884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.137871    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 16:19:54.138896 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.138885    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 16:19:54.142562 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142548    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 16:19:54.142718 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142708    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 16:19:54.142756 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142743    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 16:19:54.142785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142756    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 16:19:54.142785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142771    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 16:19:54.142785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142780    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 07 16:19:54.142869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.142790    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 16:19:54.143192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.143171    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 16:19:54.143192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.143197    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 16:19:54.145445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.145427    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 16:19:54.147558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.147544    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 16:19:54.147598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.147575    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 16:19:54.148319 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148305    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148326    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148334    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148339    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148347    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148352    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148365    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148370    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 16:19:54.148373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148376    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 16:19:54.148570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148382    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 16:19:54.148570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148391    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 16:19:54.148570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.148396    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 16:19:54.149579 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.149569    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 16:19:54.149579 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.149578    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 16:19:54.150286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.150264    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-km4xf"
Sep 07 16:19:54.152011 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.151987    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 16:19:54.152098 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.152076    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 16:19:54.152181 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.152150    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 16:19:54.153624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.153611    2577 server.go:1268] "Started kubelet"
Sep 07 16:19:54.153778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.153697    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 16:19:54.153828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.153725    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 16:19:54.153828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.153822    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 16:19:54.154518 ip-10-0-129-87 systemd[1]: Started Kubernetes Kubelet.
Sep 07 16:19:54.156593 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.156579    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 16:19:54.157176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.157157    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-km4xf"
Sep 07 16:19:54.157808 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.157788    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 16:19:54.158378 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.157465    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-87.ec2.internal.18d3164c58748b86  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-87.ec2.internal,UID:ip-10-0-129-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-87.ec2.internal,},FirstTimestamp:2026-09-07 16:19:54.15357735 +0000 UTC m=+0.328220299,LastTimestamp:2026-09-07 16:19:54.15357735 +0000 UTC m=+0.328220299,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-87.ec2.internal,}"
Sep 07 16:19:54.158456 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.158410    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 16:19:54.160598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.160583    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 16:19:54.161163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.161149    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 16:19:54.161413 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.161395    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.161956 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.161927    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 16:19:54.162272 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.161960    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 16:19:54.162402 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162389    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 16:19:54.162466 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162439    2577 factory.go:55] Registering systemd factory
Sep 07 16:19:54.162466 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162457    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 07 16:19:54.162580 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162568    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 16:19:54.162634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162581    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 16:19:54.162754 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162743    2577 factory.go:153] Registering CRI-O factory
Sep 07 16:19:54.162805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162758    2577 factory.go:223] Registration of the crio container factory successfully
Sep 07 16:19:54.162895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162825    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 16:19:54.162895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162851    2577 factory.go:103] Registering Raw factory
Sep 07 16:19:54.162895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.162873    2577 manager.go:1196] Started watching for new ooms in manager
Sep 07 16:19:54.164105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.164077    2577 manager.go:349] Starting recovery of all containers
Sep 07 16:19:54.172241 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.172209    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:54.174627 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.174590    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-87.ec2.internal\" not found" node="ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.174810 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.174790    2577 manager.go:354] Recovery completed
Sep 07 16:19:54.180951 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.180938    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.182600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.182584    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.182694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.182611    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.182694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.182623    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.183538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.183524    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.184424 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.184294    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 16:19:54.184546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.184453    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 16:19:54.184585 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.184557    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 16:19:54.190038 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.190022    2577 policy_none.go:49] "None policy: Start"
Sep 07 16:19:54.190109 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.190043    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 16:19:54.190109 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.190057    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 16:19:54.190294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.190278    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.190331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.190312    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.190331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.190328    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.192192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.192174    2577 policy_none.go:47] "Start"
Sep 07 16:19:54.228285 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228267    2577 manager.go:337] "Starting Device Plugin manager"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.228303    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228313    2577 server.go:87] "Starting device plugin registration server"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228588    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228598    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228718    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228811    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.228823    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.229338    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 16:19:54.247189 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.229397    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.266293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.266239    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 16:19:54.267771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.267754    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 16:19:54.267863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.267818    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 16:19:54.267863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.267848    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 16:19:54.267957 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.267898    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 16:19:54.269602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.269581    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:54.329233 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.329127    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.330214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.330194    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.330296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.330234    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.330296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.330245    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.330296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.330272    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.337378 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.337359    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.337436 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.337387    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-87.ec2.internal\": node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.353943 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.353922    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.368573 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.368541    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"]
Sep 07 16:19:54.368640 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.368621    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.370228 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.370213    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.370280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.370243    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.370280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.370252    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.372567 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.372555    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.372715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.372700    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.372773 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.372729    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.373304 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.373283    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.373390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.373308    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.373390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.373313    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.373390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.373323    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.373390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.373331    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.373390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.373340    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.375487 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.375472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.376102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.375783    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 16:19:54.378212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.378191    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 16:19:54.378311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.378225    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 16:19:54.378311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.378240    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Sep 07 16:19:54.401460 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.401436    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-87.ec2.internal\" not found" node="ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.405993 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.405974    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-87.ec2.internal\" not found" node="ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.454571 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.454539    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.463897 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.463870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/355f495c648da04d713df1fd6e68f66c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"355f495c648da04d713df1fd6e68f66c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.463965 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.463903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/355f495c648da04d713df1fd6e68f66c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"355f495c648da04d713df1fd6e68f66c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.463965 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.463929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/115763896ffc7ec5e812873cfb64a09e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-87.ec2.internal\" (UID: \"115763896ffc7ec5e812873cfb64a09e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.555080 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.555028    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.564401 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.564370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/355f495c648da04d713df1fd6e68f66c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"355f495c648da04d713df1fd6e68f66c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.564488 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.564415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/355f495c648da04d713df1fd6e68f66c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"355f495c648da04d713df1fd6e68f66c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.564488 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.564432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/115763896ffc7ec5e812873cfb64a09e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-87.ec2.internal\" (UID: \"115763896ffc7ec5e812873cfb64a09e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.564488 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.564481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/115763896ffc7ec5e812873cfb64a09e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-87.ec2.internal\" (UID: \"115763896ffc7ec5e812873cfb64a09e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.564610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.564509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/355f495c648da04d713df1fd6e68f66c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"355f495c648da04d713df1fd6e68f66c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.564610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.564506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/355f495c648da04d713df1fd6e68f66c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"355f495c648da04d713df1fd6e68f66c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.655907 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.655865    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.703363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.703340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.708833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:54.708277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:54.756936 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.756895    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.857478 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.857453    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:54.958148 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:54.958056    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:55.009851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.009823    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:55.058595 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:55.058563    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Sep 07 16:19:55.099055 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.099030    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 16:19:55.099450 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.099197    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 16:19:55.099450 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.099200    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 16:19:55.099450 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.099262    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 16:19:55.151005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.150973    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:55.160713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.160692    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 16:19:55.161781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.161763    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:55.163930 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.163905    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 16:14:55 +0000 UTC" deadline="2028-03-08 09:50:22.837217882 +0000 UTC"
Sep 07 16:19:55.163983 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.163930    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13145h30m27.673290073s"
Sep 07 16:19:55.174114 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.174098    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 16:19:55.178231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.178215    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 16:19:55.179068 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.179055    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Sep 07 16:19:55.186514 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.186494    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 16:19:55.193251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.193236    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-662dd"
Sep 07 16:19:55.198388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.198370    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-662dd"
Sep 07 16:19:55.269542 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:55.269511    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115763896ffc7ec5e812873cfb64a09e.slice/crio-6b34824804203a3d3acbda6c189ddbd921026da996291be21a28d7971d28009c WatchSource:0}: Error finding container 6b34824804203a3d3acbda6c189ddbd921026da996291be21a28d7971d28009c: Status 404 returned error can't find the container with id 6b34824804203a3d3acbda6c189ddbd921026da996291be21a28d7971d28009c
Sep 07 16:19:55.271374 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:55.271347    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355f495c648da04d713df1fd6e68f66c.slice/crio-0cd299b84b6e5662e547cc0145220c3e5ce7578964bd4f9a50137cf4964c9776 WatchSource:0}: Error finding container 0cd299b84b6e5662e547cc0145220c3e5ce7578964bd4f9a50137cf4964c9776: Status 404 returned error can't find the container with id 0cd299b84b6e5662e547cc0145220c3e5ce7578964bd4f9a50137cf4964c9776
Sep 07 16:19:55.275319 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:55.275301    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 16:19:57.437223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.436984    2577 apiserver.go:52] "Watching apiserver"
Sep 07 16:19:57.444965 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.444939    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 16:19:57.445517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.445493    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-njkrv","openshift-ovn-kubernetes/ovnkube-node-hcbml","kube-system/global-pull-secret-syncer-n2rmj","kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh","openshift-cluster-node-tuning-operator/tuned-mcnkh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal","openshift-multus/multus-additional-cni-plugins-n4bpc","openshift-multus/network-metrics-daemon-9pgv4","openshift-network-operator/iptables-alerter-rrvhd","kube-system/konnectivity-agent-d8lft","openshift-dns/node-resolver-hsq5z","openshift-image-registry/node-ca-vprsh","openshift-multus/multus-6q2fk"]
Sep 07 16:19:57.448201 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.448159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.450349 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.450326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkxmh\""
Sep 07 16:19:57.450658 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.450639    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.450769 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.450638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.450641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.452939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.453033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.453326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.453358    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6rwh\""
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.453415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 16:19:57.453542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.453365    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 16:19:57.454537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.454513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.454990 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.454971    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.456251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.455824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:19:57.456251 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.455925    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:19:57.457791 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.457094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 16:19:57.460177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.460160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.462493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.462445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.463077 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.463057    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.463077 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.463061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 16:19:57.463376 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.463359    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4k8xn\""
Sep 07 16:19:57.463438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.463378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/044272e8-5427-435e-acbe-b6b090dde0d3-dbus\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-log-socket\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-device-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-sys-fs\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-slash\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-node-log\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-cni-netd\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-run-netns\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-cni-bin\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-ovnkube-config\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxt8\" (UniqueName: \"kubernetes.io/projected/feea7139-c805-4835-aa3c-70436ecd4ca8-kube-api-access-wmxt8\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-registration-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.464988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5s9b\" (UniqueName: \"kubernetes.io/projected/66261745-66d1-4a78-ab76-6f51ab2bbe8f-kube-api-access-d5s9b\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-etc-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-env-overrides\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feea7139-c805-4835-aa3c-70436ecd4ca8-ovn-node-metrics-cert\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-ovnkube-script-lib\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.464996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/044272e8-5427-435e-acbe-b6b090dde0d3-kubelet-config\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bb70e-a113-4727-b293-a3a369b064ab-tmp-dir\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t75z\" (UniqueName: \"kubernetes.io/projected/348bb70e-a113-4727-b293-a3a369b064ab-kube-api-access-4t75z\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-systemd-units\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-systemd\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-var-lib-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-ovn\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.465651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-socket-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/348bb70e-a113-4727-b293-a3a369b064ab-hosts-file\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-kubelet\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dkkpw\""
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465832    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.465866    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.466140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.466532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.466146    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.467358 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.467340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.468219 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.468200    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.468448 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.468430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.468610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.468592    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6z7w5\""
Sep 07 16:19:57.468715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.468658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 16:19:57.469908 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.469890    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 16:19:57.470005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.469911    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 16:19:57.470193 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.470177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:57.470284 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.470237    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:19:57.470407 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.470391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6tcq\""
Sep 07 16:19:57.472455 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.472436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.475039 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.475015    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 16:19:57.475405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.475388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.475477 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.475417    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-drkrr\""
Sep 07 16:19:57.475537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.475388    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.475811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.475796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.477425 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.477410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.477952 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.477843    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 16:19:57.477952 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.477854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 16:19:57.477952 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.477906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nvr2w\""
Sep 07 16:19:57.479567 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.479551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 16:19:57.479840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.479822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rt544\""
Sep 07 16:19:57.479946 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.479851    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 16:19:57.494712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.494691    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 16:14:56 +0000 UTC" deadline="2028-06-20 06:51:32.57128077 +0000 UTC"
Sep 07 16:19:57.494712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.494711    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15638h31m35.076572331s"
Sep 07 16:19:57.557167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.557140    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 16:19:57.565497 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-k8s-cni-cncf-io\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.565608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-run\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.565608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/044272e8-5427-435e-acbe-b6b090dde0d3-kubelet-config\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.565608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-system-cni-dir\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.565608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysconfig\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.565608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-var-lib-kubelet\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-systemd-units\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-var-lib-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-var-lib-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-ovn\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-systemd-units\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-ovn\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/044272e8-5427-435e-acbe-b6b090dde0d3-kubelet-config\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-socket-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-multus-certs\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddkh\" (UniqueName: \"kubernetes.io/projected/c890a8e3-57f5-402f-86cb-460f4d946769-kube-api-access-mddkh\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-serviceca\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.565878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-kubelet\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/044272e8-5427-435e-acbe-b6b090dde0d3-dbus\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-kubelet\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-socket-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-netns\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-etc-kubernetes\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.565982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-device-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/044272e8-5427-435e-acbe-b6b090dde0d3-dbus\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-sys-fs\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-cni-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-device-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-sys-fs\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysctl-d\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-node-log\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-cni-netd\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.566449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c890a8e3-57f5-402f-86cb-460f4d946769-multus-daemon-config\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-node-log\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-run-netns\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-cni-bin\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-run-netns\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-ovnkube-config\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-cni-netd\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-cni-bin\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-kubelet\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cni-binary-copy\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19ece502-a089-48d7-b9fd-6c4a22ba9ab0-agent-certs\") pod \"konnectivity-agent-d8lft\" (UID: \"19ece502-a089-48d7-b9fd-6c4a22ba9ab0\") " pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-registration-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5s9b\" (UniqueName: \"kubernetes.io/projected/66261745-66d1-4a78-ab76-6f51ab2bbe8f-kube-api-access-d5s9b\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-cni-multus\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-kubernetes\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feea7139-c805-4835-aa3c-70436ecd4ca8-ovn-node-metrics-cert\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:57.567173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn22\" (UniqueName: \"kubernetes.io/projected/15b1d5eb-c920-4f26-8732-93ac130b7ca3-kube-api-access-tjn22\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-modprobe-d\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-systemd\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-host\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-registration-dir\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-ovnkube-script-lib\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.566796    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckspp\" (UniqueName: \"kubernetes.io/projected/55f00fc3-8fcf-4f2c-b951-15e809f38527-kube-api-access-ckspp\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-ovnkube-config\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.566859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bb70e-a113-4727-b293-a3a369b064ab-tmp-dir\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66261745-66d1-4a78-ab76-6f51ab2bbe8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.567060    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:58.067041727 +0000 UTC m=+2.948578676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t75z\" (UniqueName: \"kubernetes.io/projected/348bb70e-a113-4727-b293-a3a369b064ab-kube-api-access-4t75z\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-systemd\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c890a8e3-57f5-402f-86cb-460f4d946769-cni-binary-copy\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.567868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-conf-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjw94\" (UniqueName: \"kubernetes.io/projected/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-kube-api-access-pjw94\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-ovnkube-script-lib\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567284    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-systemd\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/348bb70e-a113-4727-b293-a3a369b064ab-hosts-file\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cnibin\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bb70e-a113-4727-b293-a3a369b064ab-tmp-dir\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysctl-conf\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/348bb70e-a113-4727-b293-a3a369b064ab-hosts-file\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-log-socket\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-socket-dir-parent\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28563eff-62eb-4596-a3d2-e943dd649718-iptables-alerter-script\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-log-socket\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.568490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2lx\" (UniqueName: \"kubernetes.io/projected/28563eff-62eb-4596-a3d2-e943dd649718-kube-api-access-qp2lx\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-lib-modules\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-slash\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-cnibin\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-slash\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-cni-bin\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-host\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19ece502-a089-48d7-b9fd-6c4a22ba9ab0-konnectivity-ca\") pod \"konnectivity-agent-d8lft\" (UID: \"19ece502-a089-48d7-b9fd-6c4a22ba9ab0\") " pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d902e5e-3d21-4777-a390-8a2649b7c381-tmp\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28563eff-62eb-4596-a3d2-e943dd649718-host-slash\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flk5h\" (UniqueName: \"kubernetes.io/projected/1d902e5e-3d21-4777-a390-8a2649b7c381-kube-api-access-flk5h\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxt8\" (UniqueName: \"kubernetes.io/projected/feea7139-c805-4835-aa3c-70436ecd4ca8-kube-api-access-wmxt8\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.567990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-hostroot\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-sys\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.569359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-tuned\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-etc-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-etc-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-env-overrides\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-system-cni-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feea7139-c805-4835-aa3c-70436ecd4ca8-run-openvswitch\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feea7139-c805-4835-aa3c-70436ecd4ca8-env-overrides\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.568616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" event={"ID":"355f495c648da04d713df1fd6e68f66c","Type":"ContainerStarted","Data":"0cd299b84b6e5662e547cc0145220c3e5ce7578964bd4f9a50137cf4964c9776"}
Sep 07 16:19:57.570117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.570113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal" event={"ID":"115763896ffc7ec5e812873cfb64a09e","Type":"ContainerStarted","Data":"6b34824804203a3d3acbda6c189ddbd921026da996291be21a28d7971d28009c"}
Sep 07 16:19:57.571573 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.571552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feea7139-c805-4835-aa3c-70436ecd4ca8-ovn-node-metrics-cert\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.572804 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.572787    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:19:57.572895 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.572810    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:19:57.572895 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.572826    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:19:57.572895 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.572892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:58.072876599 +0000 UTC m=+2.954413565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:19:57.575377 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.575354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5s9b\" (UniqueName: \"kubernetes.io/projected/66261745-66d1-4a78-ab76-6f51ab2bbe8f-kube-api-access-d5s9b\") pod \"aws-ebs-csi-driver-node-kfrrh\" (UID: \"66261745-66d1-4a78-ab76-6f51ab2bbe8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.576221 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.576204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t75z\" (UniqueName: \"kubernetes.io/projected/348bb70e-a113-4727-b293-a3a369b064ab-kube-api-access-4t75z\") pod \"node-resolver-hsq5z\" (UID: \"348bb70e-a113-4727-b293-a3a369b064ab\") " pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.576305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.576253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxt8\" (UniqueName: \"kubernetes.io/projected/feea7139-c805-4835-aa3c-70436ecd4ca8-kube-api-access-wmxt8\") pod \"ovnkube-node-hcbml\" (UID: \"feea7139-c805-4835-aa3c-70436ecd4ca8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.669045 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-modprobe-d\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-systemd\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-host\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckspp\" (UniqueName: \"kubernetes.io/projected/55f00fc3-8fcf-4f2c-b951-15e809f38527-kube-api-access-ckspp\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c890a8e3-57f5-402f-86cb-460f4d946769-cni-binary-copy\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-conf-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.669188    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:19:57.669214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjw94\" (UniqueName: \"kubernetes.io/projected/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-kube-api-access-pjw94\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.669446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-modprobe-d\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cnibin\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.669446 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:57.669256    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:58.169235514 +0000 UTC m=+3.050772464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:19:57.669446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cnibin\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.669446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-systemd\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-host\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-conf-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysctl-conf\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-socket-dir-parent\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28563eff-62eb-4596-a3d2-e943dd649718-iptables-alerter-script\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2lx\" (UniqueName: \"kubernetes.io/projected/28563eff-62eb-4596-a3d2-e943dd649718-kube-api-access-qp2lx\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.669654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-lib-modules\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-cnibin\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-cni-bin\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-host\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19ece502-a089-48d7-b9fd-6c4a22ba9ab0-konnectivity-ca\") pod \"konnectivity-agent-d8lft\" (UID: \"19ece502-a089-48d7-b9fd-6c4a22ba9ab0\") " pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d902e5e-3d21-4777-a390-8a2649b7c381-tmp\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28563eff-62eb-4596-a3d2-e943dd649718-host-slash\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk5h\" (UniqueName: \"kubernetes.io/projected/1d902e5e-3d21-4777-a390-8a2649b7c381-kube-api-access-flk5h\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-cnibin\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-hostroot\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-cni-bin\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.669924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-sys\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-socket-dir-parent\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysctl-conf\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-host\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.669982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-tuned\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-system-cni-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-k8s-cni-cncf-io\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-sys\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-run\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-k8s-cni-cncf-io\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-system-cni-dir\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-system-cni-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b1d5eb-c920-4f26-8732-93ac130b7ca3-system-cni-dir\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-run\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-lib-modules\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.670517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-hostroot\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28563eff-62eb-4596-a3d2-e943dd649718-host-slash\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysconfig\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-var-lib-kubelet\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysconfig\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-multus-certs\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddkh\" (UniqueName: \"kubernetes.io/projected/c890a8e3-57f5-402f-86cb-460f4d946769-kube-api-access-mddkh\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-serviceca\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-netns\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-etc-kubernetes\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-cni-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-multus-certs\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28563eff-62eb-4596-a3d2-e943dd649718-iptables-alerter-script\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19ece502-a089-48d7-b9fd-6c4a22ba9ab0-konnectivity-ca\") pod \"konnectivity-agent-d8lft\" (UID: \"19ece502-a089-48d7-b9fd-6c4a22ba9ab0\") " pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysctl-d\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c890a8e3-57f5-402f-86cb-460f4d946769-multus-daemon-config\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-sysctl-d\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.671445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.670984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-var-lib-kubelet\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-run-netns\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-kubelet\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-multus-cni-dir\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-etc-kubernetes\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-kubelet\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-serviceca\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cni-binary-copy\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19ece502-a089-48d7-b9fd-6c4a22ba9ab0-agent-certs\") pod \"konnectivity-agent-d8lft\" (UID: \"19ece502-a089-48d7-b9fd-6c4a22ba9ab0\") " pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-cni-multus\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-kubernetes\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn22\" (UniqueName: \"kubernetes.io/projected/15b1d5eb-c920-4f26-8732-93ac130b7ca3-kube-api-access-tjn22\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c890a8e3-57f5-402f-86cb-460f4d946769-host-var-lib-cni-multus\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15b1d5eb-c920-4f26-8732-93ac130b7ca3-cni-binary-copy\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.671921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-kubernetes\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.672305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.672248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c890a8e3-57f5-402f-86cb-460f4d946769-cni-binary-copy\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.672328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c890a8e3-57f5-402f-86cb-460f4d946769-multus-daemon-config\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.672909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.672705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d902e5e-3d21-4777-a390-8a2649b7c381-etc-tuned\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.673086 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.673066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d902e5e-3d21-4777-a390-8a2649b7c381-tmp\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.673898 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.673879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19ece502-a089-48d7-b9fd-6c4a22ba9ab0-agent-certs\") pod \"konnectivity-agent-d8lft\" (UID: \"19ece502-a089-48d7-b9fd-6c4a22ba9ab0\") " pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.678488 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.678463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjw94\" (UniqueName: \"kubernetes.io/projected/0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc-kube-api-access-pjw94\") pod \"node-ca-vprsh\" (UID: \"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc\") " pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.679159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.679133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2lx\" (UniqueName: \"kubernetes.io/projected/28563eff-62eb-4596-a3d2-e943dd649718-kube-api-access-qp2lx\") pod \"iptables-alerter-rrvhd\" (UID: \"28563eff-62eb-4596-a3d2-e943dd649718\") " pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.679295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.679221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddkh\" (UniqueName: \"kubernetes.io/projected/c890a8e3-57f5-402f-86cb-460f4d946769-kube-api-access-mddkh\") pod \"multus-6q2fk\" (UID: \"c890a8e3-57f5-402f-86cb-460f4d946769\") " pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.679295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.679262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckspp\" (UniqueName: \"kubernetes.io/projected/55f00fc3-8fcf-4f2c-b951-15e809f38527-kube-api-access-ckspp\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:57.679765 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.679737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flk5h\" (UniqueName: \"kubernetes.io/projected/1d902e5e-3d21-4777-a390-8a2649b7c381-kube-api-access-flk5h\") pod \"tuned-mcnkh\" (UID: \"1d902e5e-3d21-4777-a390-8a2649b7c381\") " pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.680279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.680265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn22\" (UniqueName: \"kubernetes.io/projected/15b1d5eb-c920-4f26-8732-93ac130b7ca3-kube-api-access-tjn22\") pod \"multus-additional-cni-plugins-n4bpc\" (UID: \"15b1d5eb-c920-4f26-8732-93ac130b7ca3\") " pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.696123 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.696071    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:57.761120 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.761078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hsq5z"
Sep 07 16:19:57.769296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.769274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:19:57.777633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.777611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh"
Sep 07 16:19:57.783193 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.783174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6q2fk"
Sep 07 16:19:57.789764 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.789730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vprsh"
Sep 07 16:19:57.795339 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.795325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n4bpc"
Sep 07 16:19:57.804831 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.804812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrvhd"
Sep 07 16:19:57.811420 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.811403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:19:57.816397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.816369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mcnkh"
Sep 07 16:19:57.925888 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:57.925845    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:58.073828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.073746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:19:58.073828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.073806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:58.073987 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.073909    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:19:58.073987 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.073925    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:19:58.073987 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.073949    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:19:58.073987 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.073960    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:19:58.073987 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.073974    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:59.073958795 +0000 UTC m=+3.955495740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:19:58.074130 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.073999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:59.073987288 +0000 UTC m=+3.955524234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:19:58.174653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.174617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:58.174840 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.174815    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:19:58.174913 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:58.174902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:59.174881655 +0000 UTC m=+4.056418615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:19:58.375956 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.375928    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28563eff_62eb_4596_a3d2_e943dd649718.slice/crio-28b28a3a173f6ab74a222cc7fd3d812cb7203f63ab4aaa76f4fff60c7fc1681d WatchSource:0}: Error finding container 28b28a3a173f6ab74a222cc7fd3d812cb7203f63ab4aaa76f4fff60c7fc1681d: Status 404 returned error can't find the container with id 28b28a3a173f6ab74a222cc7fd3d812cb7203f63ab4aaa76f4fff60c7fc1681d
Sep 07 16:19:58.377103 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.377075    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ece502_a089_48d7_b9fd_6c4a22ba9ab0.slice/crio-e29bc4ae47f390ae03a6d83fecfa37e6329efa8f02947f7179e9927939f535cd WatchSource:0}: Error finding container e29bc4ae47f390ae03a6d83fecfa37e6329efa8f02947f7179e9927939f535cd: Status 404 returned error can't find the container with id e29bc4ae47f390ae03a6d83fecfa37e6329efa8f02947f7179e9927939f535cd
Sep 07 16:19:58.378863 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.378839    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f93a8e0_8f59_41cd_acd2_8e4baebfa6dc.slice/crio-f4da0e14c437126c24c682ecae73e6c5da1a1826cb449bda4fc1a48a5b743871 WatchSource:0}: Error finding container f4da0e14c437126c24c682ecae73e6c5da1a1826cb449bda4fc1a48a5b743871: Status 404 returned error can't find the container with id f4da0e14c437126c24c682ecae73e6c5da1a1826cb449bda4fc1a48a5b743871
Sep 07 16:19:58.401101 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.401071    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66261745_66d1_4a78_ab76_6f51ab2bbe8f.slice/crio-1f0c0681e9cad25309010ab24a5debe133b70f61ca863f5824482cfd8b12e0db WatchSource:0}: Error finding container 1f0c0681e9cad25309010ab24a5debe133b70f61ca863f5824482cfd8b12e0db: Status 404 returned error can't find the container with id 1f0c0681e9cad25309010ab24a5debe133b70f61ca863f5824482cfd8b12e0db
Sep 07 16:19:58.401788 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.401650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348bb70e_a113_4727_b293_a3a369b064ab.slice/crio-d97e16ed716e0e6d25080bbbee3307decee9aa12c9112a2ae7531641c0de2530 WatchSource:0}: Error finding container d97e16ed716e0e6d25080bbbee3307decee9aa12c9112a2ae7531641c0de2530: Status 404 returned error can't find the container with id d97e16ed716e0e6d25080bbbee3307decee9aa12c9112a2ae7531641c0de2530
Sep 07 16:19:58.402510 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.402491    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b1d5eb_c920_4f26_8732_93ac130b7ca3.slice/crio-6f75cbb2630ecd1b1cd3dacc486e8f7aff9af80ad1c501f5d75474de240d12a6 WatchSource:0}: Error finding container 6f75cbb2630ecd1b1cd3dacc486e8f7aff9af80ad1c501f5d75474de240d12a6: Status 404 returned error can't find the container with id 6f75cbb2630ecd1b1cd3dacc486e8f7aff9af80ad1c501f5d75474de240d12a6
Sep 07 16:19:58.408069 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:19:58.408014    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeea7139_c805_4835_aa3c_70436ecd4ca8.slice/crio-8913f8585e5f81e1f99d108ad83d8bcae16f7e7f0fd1aacb9fb6c557fd660501 WatchSource:0}: Error finding container 8913f8585e5f81e1f99d108ad83d8bcae16f7e7f0fd1aacb9fb6c557fd660501: Status 404 returned error can't find the container with id 8913f8585e5f81e1f99d108ad83d8bcae16f7e7f0fd1aacb9fb6c557fd660501
Sep 07 16:19:58.495103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.495052    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 16:14:56 +0000 UTC" deadline="2028-05-18 01:10:12.005167702 +0000 UTC"
Sep 07 16:19:58.495103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.495098    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14840h50m13.510072505s"
Sep 07 16:19:58.573950 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.573907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6q2fk" event={"ID":"c890a8e3-57f5-402f-86cb-460f4d946769","Type":"ContainerStarted","Data":"872aecfadf19556549dc8147738096a3dea78870033ac4c4bd01032ba8488265"}
Sep 07 16:19:58.575122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.575095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mcnkh" event={"ID":"1d902e5e-3d21-4777-a390-8a2649b7c381","Type":"ContainerStarted","Data":"4531c0ec0f68be08847234f453f7c0a95158cfd190c743968234dc17f011ae50"}
Sep 07 16:19:58.576636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.576612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh" event={"ID":"66261745-66d1-4a78-ab76-6f51ab2bbe8f","Type":"ContainerStarted","Data":"1f0c0681e9cad25309010ab24a5debe133b70f61ca863f5824482cfd8b12e0db"}
Sep 07 16:19:58.578050 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.578023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d8lft" event={"ID":"19ece502-a089-48d7-b9fd-6c4a22ba9ab0","Type":"ContainerStarted","Data":"e29bc4ae47f390ae03a6d83fecfa37e6329efa8f02947f7179e9927939f535cd"}
Sep 07 16:19:58.579406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.579384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrvhd" event={"ID":"28563eff-62eb-4596-a3d2-e943dd649718","Type":"ContainerStarted","Data":"28b28a3a173f6ab74a222cc7fd3d812cb7203f63ab4aaa76f4fff60c7fc1681d"}
Sep 07 16:19:58.580619 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.580598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"8913f8585e5f81e1f99d108ad83d8bcae16f7e7f0fd1aacb9fb6c557fd660501"}
Sep 07 16:19:58.581853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.581826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerStarted","Data":"6f75cbb2630ecd1b1cd3dacc486e8f7aff9af80ad1c501f5d75474de240d12a6"}
Sep 07 16:19:58.582994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.582976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hsq5z" event={"ID":"348bb70e-a113-4727-b293-a3a369b064ab","Type":"ContainerStarted","Data":"d97e16ed716e0e6d25080bbbee3307decee9aa12c9112a2ae7531641c0de2530"}
Sep 07 16:19:58.584131 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.584109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vprsh" event={"ID":"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc","Type":"ContainerStarted","Data":"f4da0e14c437126c24c682ecae73e6c5da1a1826cb449bda4fc1a48a5b743871"}
Sep 07 16:19:58.586055 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.586032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal" event={"ID":"115763896ffc7ec5e812873cfb64a09e","Type":"ContainerStarted","Data":"8e2285768abb24d0374190920fb246f414db764d507665429c3fc7835604c373"}
Sep 07 16:19:58.598814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:58.598776    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal" podStartSLOduration=2.598737673 podStartE2EDuration="2.598737673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:58.598582417 +0000 UTC m=+3.480119385" watchObservedRunningTime="2026-09-07 16:19:58.598737673 +0000 UTC m=+3.480274637"
Sep 07 16:19:59.018108 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.018072    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 16:19:59.081500 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.081457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:19:59.081687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.081515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:59.081687 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.081643    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:19:59.081687 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.081648    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:19:59.081687 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.081679    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:19:59.081687 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.081692    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:19:59.081961 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.081707    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:01.08168844 +0000 UTC m=+5.963225402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:19:59.081961 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.081763    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:20:01.081728396 +0000 UTC m=+5.963265366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:19:59.181934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.181894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:59.182112 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.182045    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:19:59.182112 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.182111    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:01.182093692 +0000 UTC m=+6.063630645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:19:59.563667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.562881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:19:59.563667 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.563021    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:19:59.563667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.563401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:19:59.563667 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.563482    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:19:59.563667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.563555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:19:59.563667 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:19:59.563622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:19:59.619156 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.618673    2577 generic.go:356] "Generic (PLEG): container finished" podID="355f495c648da04d713df1fd6e68f66c" containerID="c3951ce10a63c6682d80c432ef55bf879eb9fa85194c93e0baa8b6d2bb01688d" exitCode=0
Sep 07 16:19:59.619327 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:19:59.619277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" event={"ID":"355f495c648da04d713df1fd6e68f66c","Type":"ContainerDied","Data":"c3951ce10a63c6682d80c432ef55bf879eb9fa85194c93e0baa8b6d2bb01688d"}
Sep 07 16:20:00.644672 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:00.643508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" event={"ID":"355f495c648da04d713df1fd6e68f66c","Type":"ContainerStarted","Data":"f29fc8263c311136dd987173ac9e8bdcc7dc81e16843c166b7c3be01cbe8a25e"}
Sep 07 16:20:01.107380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:01.107291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:01.107380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:01.107358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:01.107650 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.107535    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:01.107650 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.107601    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:05.107582728 +0000 UTC m=+9.989119689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:01.108061 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.108040    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:20:01.108174 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.108067    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:20:01.108174 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.108079    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:01.108174 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.108126    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:20:05.108111349 +0000 UTC m=+9.989648309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:01.208247 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:01.208200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:01.208424 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.208385    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:01.208506 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.208453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:05.208435341 +0000 UTC m=+10.089972291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:01.564188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:01.561987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:01.564188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:01.561987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:01.564188 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.562124    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:01.564188 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.562228    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:01.564188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:01.561993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:01.564188 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:01.562327    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:03.562022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:03.561986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:03.562425 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:03.562112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:03.562425 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:03.562124    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:03.562425 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:03.561995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:03.562425 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:03.562322    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:03.562570 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:03.562432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:05.143085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:05.143131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.143237    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.143251    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.143269    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.143282    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.143296    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:13.143278697 +0000 UTC m=+18.024815661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:05.143584 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.143315    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:20:13.143305051 +0000 UTC m=+18.024841997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:05.244151 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:05.244113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:05.244350 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.244273    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:05.244411 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.244359    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:13.244321283 +0000 UTC m=+18.125858235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:05.563907 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:05.563823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:05.564038 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.563944    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:05.564324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:05.564305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:05.564423 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.564403    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:05.564479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:05.564467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:05.564570 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:05.564554    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:07.561343 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:07.561308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:07.561806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:07.561321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:07.561806 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:07.561448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:07.561806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:07.561483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:07.561806 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:07.561574    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:07.561806 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:07.561656    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:09.561263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:09.561233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:09.561263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:09.561259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:09.561263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:09.561226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:09.561815 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:09.561375    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:09.561815 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:09.561413    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:09.561815 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:09.561531    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:11.562129 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:11.562099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:11.562129 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:11.562133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:11.562565 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:11.562098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:11.562565 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:11.562232    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:11.562565 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:11.562289    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:11.562565 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:11.562383    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:13.203150 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:13.203111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:13.203150 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:13.203160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:13.203743 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.203255    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:13.203743 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.203284    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:20:13.203743 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.203314    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:20:13.203743 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.203320    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:29.203305936 +0000 UTC m=+34.084842904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:13.203743 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.203328    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:13.203743 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.203375    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:20:29.203360034 +0000 UTC m=+34.084896981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:13.303853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:13.303821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:13.304027 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.303945    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:13.304027 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.303999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:29.303985315 +0000 UTC m=+34.185522274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:13.561486 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:13.561399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:13.561635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:13.561399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:13.561635 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.561555    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:13.561635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:13.561399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:13.561813 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.561642    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:13.561813 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:13.561691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:15.562091 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:15.562053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:15.562482 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:15.562172    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:15.562482 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:15.562185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:15.562482 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:15.562297    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:15.562482 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:15.562329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:15.562482 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:15.562404    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:16.679003 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.678779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"6dcc9860e6428fbe2b3f936ca64dd32d8769444676d143350dfdc0033a2b510d"}
Sep 07 16:20:16.679589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.679093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"f2dc38dff69ed139c1212bcd088db6ea7a401ef518dfc0dfcdbe615c32228d11"}
Sep 07 16:20:16.679589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.679152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"6b27f2a2a76df0493e1abc5c8e99e45c1ef22357e3fb4073568804de2af43668"}
Sep 07 16:20:16.679589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.679200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"b2527765f00f8ac2a2453a67c609b2e232eb69cfa56f6a70216d1e720381ef79"}
Sep 07 16:20:16.679589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.679248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"0249df846ecebff71ca6718bb73478773575fcf6b3a86a611cdfa0751293e89a"}
Sep 07 16:20:16.680372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.680348    2577 generic.go:356] "Generic (PLEG): container finished" podID="15b1d5eb-c920-4f26-8732-93ac130b7ca3" containerID="0d3320bf9f547bd5ec610e7aae85e215c63c24d3e669d81e43a983278bb8621d" exitCode=0
Sep 07 16:20:16.680486 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.680438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerDied","Data":"0d3320bf9f547bd5ec610e7aae85e215c63c24d3e669d81e43a983278bb8621d"}
Sep 07 16:20:16.681973 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.681947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hsq5z" event={"ID":"348bb70e-a113-4727-b293-a3a369b064ab","Type":"ContainerStarted","Data":"24650215ed074e1feac9bc041668c124da1fba0d1005f435f9c0a27320de51dd"}
Sep 07 16:20:16.683521 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.683499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vprsh" event={"ID":"0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc","Type":"ContainerStarted","Data":"11850aa6a950380e1b87317d174eb0c706cd4c7b142ea76a8e09cc81abefae11"}
Sep 07 16:20:16.685920 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.685893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6q2fk" event={"ID":"c890a8e3-57f5-402f-86cb-460f4d946769","Type":"ContainerStarted","Data":"5582e06f7a4899f0da1d8465c06a67582b0e2116c3d458cd72e1cd0e63817896"}
Sep 07 16:20:16.688030 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.688006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mcnkh" event={"ID":"1d902e5e-3d21-4777-a390-8a2649b7c381","Type":"ContainerStarted","Data":"0b49d024e5a27bf0e85ee401cb2fd704b6693fe20cf4eb6214837e53c9976f7f"}
Sep 07 16:20:16.689373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.689340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh" event={"ID":"66261745-66d1-4a78-ab76-6f51ab2bbe8f","Type":"ContainerStarted","Data":"185928dd4ae118d563e3fd183b11260d25c3745ade6fca280c426f86df9b6436"}
Sep 07 16:20:16.690964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.690943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d8lft" event={"ID":"19ece502-a089-48d7-b9fd-6c4a22ba9ab0","Type":"ContainerStarted","Data":"f80ba91c0cecaa7b13bb136bf640adc973a9710362a678a5388c0aeb27d0e46c"}
Sep 07 16:20:16.700509 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.700459    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" podStartSLOduration=20.700447773 podStartE2EDuration="20.700447773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:00.658730277 +0000 UTC m=+5.540267247" watchObservedRunningTime="2026-09-07 16:20:16.700447773 +0000 UTC m=+21.581984740"
Sep 07 16:20:16.719191 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.716789    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6q2fk" podStartSLOduration=4.241894779 podStartE2EDuration="21.716771169s" totalImagesPullingTime="17.47487639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:16.715531886 +0000 UTC m=+21.597068849" watchObservedRunningTime="2026-09-07 16:20:16.716771169 +0000 UTC m=+21.598308130"
Sep 07 16:20:16.731520 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.731483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vprsh" podStartSLOduration=4.385253557 podStartE2EDuration="21.7314703s" totalImagesPullingTime="17.346216743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:16.731406171 +0000 UTC m=+21.612943138" watchObservedRunningTime="2026-09-07 16:20:16.7314703 +0000 UTC m=+21.613007266"
Sep 07 16:20:16.744158 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.744126    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hsq5z" podStartSLOduration=4.40464315 podStartE2EDuration="21.744114686s" totalImagesPullingTime="17.339471536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:16.743856724 +0000 UTC m=+21.625393690" watchObservedRunningTime="2026-09-07 16:20:16.744114686 +0000 UTC m=+21.625651653"
Sep 07 16:20:16.756610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.756572    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d8lft" podStartSLOduration=11.539145785 podStartE2EDuration="21.756559809s" totalImagesPullingTime="10.217414024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:16.756457104 +0000 UTC m=+21.637994071" watchObservedRunningTime="2026-09-07 16:20:16.756559809 +0000 UTC m=+21.638096776"
Sep 07 16:20:16.771267 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:16.771229    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mcnkh" podStartSLOduration=3.294635211 podStartE2EDuration="20.771216463s" totalImagesPullingTime="17.476581252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:16.770801659 +0000 UTC m=+21.652338626" watchObservedRunningTime="2026-09-07 16:20:16.771216463 +0000 UTC m=+21.652753431"
Sep 07 16:20:17.106670 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.106517    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 16:20:17.160202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.160162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:17.534320 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.534219    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T16:20:17.106667144Z","UUID":"bd6b2742-bbb9-4e62-9db7-c9385c0568ce","Handler":null,"Name":"","Endpoint":""}
Sep 07 16:20:17.535884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.535861    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 16:20:17.535884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.535896    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 16:20:17.549555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.549529    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:20:17.563773 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.563732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:17.563909 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:17.563857    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:17.563988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.563976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:17.564076 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:17.564056    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:17.564154 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.564141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:17.564256 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:17.564238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:17.696231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.696179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh" event={"ID":"66261745-66d1-4a78-ab76-6f51ab2bbe8f","Type":"ContainerStarted","Data":"17630e6701a123af4263bfb7a5dd0111826e10bb5b81a16470a7f42bd3130a66"}
Sep 07 16:20:17.698117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.698078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrvhd" event={"ID":"28563eff-62eb-4596-a3d2-e943dd649718","Type":"ContainerStarted","Data":"754097c8e8b5e07f8dfbc9c99c5298fad35c1993b55ffb4e45e883bda0d3a79a"}
Sep 07 16:20:17.703633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.703588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"ac4a6a82e3d5566a08f77653f8bdb42fc7dbe613a6e1525ae46f046ca627af1e"}
Sep 07 16:20:17.712486 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.712443    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rrvhd" podStartSLOduration=5.366867319 podStartE2EDuration="22.712430556s" totalImagesPullingTime="17.345563237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:17.712402628 +0000 UTC m=+22.593939596" watchObservedRunningTime="2026-09-07 16:20:17.712430556 +0000 UTC m=+22.593967524"
Sep 07 16:20:17.965886 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:17.965855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:18.208827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:18.208785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:18.368985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:18.368906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:18.569865 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:18.569830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:18.710582 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:18.710542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh" event={"ID":"66261745-66d1-4a78-ab76-6f51ab2bbe8f","Type":"ContainerStarted","Data":"3e16efde4755a4415500c92594ebf11c8156b0eee73032d19dd89d94cb318ed6"}
Sep 07 16:20:18.727990 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:18.727939    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfrrh" podStartSLOduration=4.5625766599999995 podStartE2EDuration="23.727924454s" totalImagesPullingTime="19.165347794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:18.727450122 +0000 UTC m=+23.608987090" watchObservedRunningTime="2026-09-07 16:20:18.727924454 +0000 UTC m=+23.609461418"
Sep 07 16:20:18.765408 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:18.765361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:19.164546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.164511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:19.561643 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.561566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:19.561828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.561571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:19.561828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.561696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:19.561828 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:19.561696    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:19.561828 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:19.561804    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:19.562003 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:19.561902    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:19.566594 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.566572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:19.667617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.667587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:19.716637 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.716600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"b69c0806d91e4c78a730c2fd12e39f58cfdcc3040363e33b615ee8dbd7453db7"}
Sep 07 16:20:19.962851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:19.962816    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:20.364202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:20.364111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:20.560385 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:20.560350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:20.663182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:20.663145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:20.767927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:20.767901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:20.974215 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:20.974182    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:21.042607 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.042435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:20:21.043047 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.043027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:20:21.562108 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.562075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:21.562278 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.562081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:21.562278 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:21.562184    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:21.562353 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:21.562291    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:21.562353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.562081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:21.562421 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:21.562387    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:21.654663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.654632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:21.708393 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.708356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:21.724549 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.724517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" event={"ID":"feea7139-c805-4835-aa3c-70436ecd4ca8","Type":"ContainerStarted","Data":"3ac790d58b70bde8f0431a137ab32c5d862d5aac0a1920b276b4ce3ce8ad603c"}
Sep 07 16:20:21.724938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.724905    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:20:21.730012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.729969    2577 generic.go:356] "Generic (PLEG): container finished" podID="15b1d5eb-c920-4f26-8732-93ac130b7ca3" containerID="6878b7ef7f96da714b49335624d45a48b351a01c963ff8bd4e55b2b03fe8c58b" exitCode=0
Sep 07 16:20:21.730133 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.730055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerDied","Data":"6878b7ef7f96da714b49335624d45a48b351a01c963ff8bd4e55b2b03fe8c58b"}
Sep 07 16:20:21.730849 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.730729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d8lft"
Sep 07 16:20:21.741941 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.741921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:20:21.753606 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:21.753557    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml" podStartSLOduration=9.031631376 podStartE2EDuration="26.75354295s" totalImagesPullingTime="17.721911574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:21.753083399 +0000 UTC m=+26.634620366" watchObservedRunningTime="2026-09-07 16:20:21.75354295 +0000 UTC m=+26.635079897"
Sep 07 16:20:22.055393 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.055366    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:22.160946 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.160914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:22.365445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.365384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:22.564189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.564152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:22.733991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.733962    2577 generic.go:356] "Generic (PLEG): container finished" podID="15b1d5eb-c920-4f26-8732-93ac130b7ca3" containerID="5aa9fdb8f0dc4357bbf52483c3f4510757c470b146edb234911fcde3c64b2e43" exitCode=0
Sep 07 16:20:22.734121 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.734056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerDied","Data":"5aa9fdb8f0dc4357bbf52483c3f4510757c470b146edb234911fcde3c64b2e43"}
Sep 07 16:20:22.734715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.734639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:20:22.734798 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.734723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:20:22.749595 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.749576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:20:22.767255 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.767234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:22.963763 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:22.963729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:23.561839 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:23.561791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:23.561839 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:23.561828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:23.562406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:23.561802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:23.562406 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:23.561926    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:23.562406 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:23.562022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:23.562406 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:23.562123    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:23.738538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:23.738503    2577 generic.go:356] "Generic (PLEG): container finished" podID="15b1d5eb-c920-4f26-8732-93ac130b7ca3" containerID="9e4391e8bba746a43f59bba167a421a52e4d152c42152c3f7404947c4a15d7d4" exitCode=0
Sep 07 16:20:23.738706 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:23.738546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerDied","Data":"9e4391e8bba746a43f59bba167a421a52e4d152c42152c3f7404947c4a15d7d4"}
Sep 07 16:20:23.766065 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:23.766037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:24.167831 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:24.167799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:24.363300 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:24.363232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:24.964930 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:24.964903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:25.364419 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:25.364325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:25.563375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:25.563312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:25.563693 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:25.563666    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:25.564642 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:25.564620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:25.564873 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:25.564853    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:25.565106 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:25.565091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:25.565342 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:25.565320    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:25.764171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:25.764128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:25.966972 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:25.966933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:26.366160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:26.365948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:26.567099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:26.567070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:27.561683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:27.561646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:27.562128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:27.561653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:27.562128 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:27.561814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:27.562128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:27.561649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:27.562128 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:27.561870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:27.562128 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:27.561907    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:28.753621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:28.753584    2577 generic.go:356] "Generic (PLEG): container finished" podID="15b1d5eb-c920-4f26-8732-93ac130b7ca3" containerID="0235dfad1ac9ac5bdbb668decc5dcec8c9bbe4f0e3fa52b3791e03d193ce7067" exitCode=0
Sep 07 16:20:28.754068 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:28.753659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerDied","Data":"0235dfad1ac9ac5bdbb668decc5dcec8c9bbe4f0e3fa52b3791e03d193ce7067"}
Sep 07 16:20:29.227821 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.227780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.227837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.227928    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.227945    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.227968    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.227977    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p2j9d for pod openshift-network-diagnostics/network-check-target-njkrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.227988    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret podName:044272e8-5427-435e-acbe-b6b090dde0d3 nodeName:}" failed. No retries permitted until 2026-09-07 16:21:01.227970538 +0000 UTC m=+66.109507501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret") pod "global-pull-secret-syncer-n2rmj" (UID: "044272e8-5427-435e-acbe-b6b090dde0d3") : object "kube-system"/"original-pull-secret" not registered
Sep 07 16:20:29.228017 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.228018    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d podName:4935ae52-b9c9-40da-9482-714b29e4048e nodeName:}" failed. No retries permitted until 2026-09-07 16:21:01.228006084 +0000 UTC m=+66.109543043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2j9d" (UniqueName: "kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d") pod "network-check-target-njkrv" (UID: "4935ae52-b9c9-40da-9482-714b29e4048e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 16:20:29.329264 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.329228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:29.329421 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.329348    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:29.329421 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.329401    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs podName:55f00fc3-8fcf-4f2c-b951-15e809f38527 nodeName:}" failed. No retries permitted until 2026-09-07 16:21:01.329387726 +0000 UTC m=+66.210924676 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs") pod "network-metrics-daemon-9pgv4" (UID: "55f00fc3-8fcf-4f2c-b951-15e809f38527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 16:20:29.562067 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.561987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:29.562204 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.562087    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:29.562204 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.562096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:29.562278 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.562203    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:29.562278 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.562256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:29.562337 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:29.562317    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:29.759769 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.759719    2577 generic.go:356] "Generic (PLEG): container finished" podID="15b1d5eb-c920-4f26-8732-93ac130b7ca3" containerID="5b0b0fec83a845797c71b740747736d99f0a473105c6392b8b7d9bfd31a42c84" exitCode=0
Sep 07 16:20:29.760153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:29.759780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerDied","Data":"5b0b0fec83a845797c71b740747736d99f0a473105c6392b8b7d9bfd31a42c84"}
Sep 07 16:20:30.233248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.233210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:30.333976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.333943    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:30.393632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.393605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:30.704727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.704688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:30.765407 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.765367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" event={"ID":"15b1d5eb-c920-4f26-8732-93ac130b7ca3","Type":"ContainerStarted","Data":"00dd440f78662294cb5085685ae9d47c472dd85fa4ce0f9c10278d4c5a499d0a"}
Sep 07 16:20:30.785170 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.785121    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n4bpc" podStartSLOduration=8.6165106 podStartE2EDuration="35.785104927s" totalImagesPullingTime="27.168594327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:30.78366021 +0000 UTC m=+35.665197178" watchObservedRunningTime="2026-09-07 16:20:30.785104927 +0000 UTC m=+35.666641920"
Sep 07 16:20:30.890256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.890204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:30.947619 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:30.947588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:31.289788 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.289743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:31.352790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.352737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:31.562028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.561945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:31.562028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.561980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:31.562238 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.561949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:31.562238 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:31.562049    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:31.562238 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:31.562127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:31.562238 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:31.562193    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:31.686528 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.686496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:31.742088 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:31.742060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:32.111598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:32.111563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:32.308448 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:32.308420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:32.718537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:32.718505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:33.528288 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:33.528252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:33.561615 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:33.561583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:33.561615 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:33.561603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:33.561846 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:33.561583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:33.561846 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:33.561710    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:33.561846 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:33.561831    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:33.561998 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:33.561938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:33.577841 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:33.577810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:34.274102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:34.274065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:34.426067 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:34.426037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:34.574875 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:34.574792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:34.675212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:34.675179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:34.771646 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:34.771618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:34.901830 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:34.901802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:35.234942 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.234911    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n2rmj"]
Sep 07 16:20:35.235099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.235052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:35.235149 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:35.235133    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:35.238903 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.238872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njkrv"]
Sep 07 16:20:35.239039 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.238930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9pgv4"]
Sep 07 16:20:35.239039 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.239000    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:35.239137 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:35.239102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:35.239304 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.239287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:35.239402 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:35.239377    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:35.534689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.534608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:35.843251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.843017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:35.897241 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:35.897209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:36.561124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:36.561093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:36.561275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:36.561101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:36.561275 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:36.561200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:36.561275 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:36.561255    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:36.561275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:36.561107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:36.561396 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:36.561311    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:36.650176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:36.650149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:36.699526 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:36.699500    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:37.792701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:37.792615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:37.844091 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:37.844059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:38.561444 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:38.561409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:38.561635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:38.561457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:38.561635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:38.561512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:38.561635 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:38.561526    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pgv4" podUID="55f00fc3-8fcf-4f2c-b951-15e809f38527"
Sep 07 16:20:38.561635 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:38.561577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njkrv" podUID="4935ae52-b9c9-40da-9482-714b29e4048e"
Sep 07 16:20:38.561840 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:38.561661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2rmj" podUID="044272e8-5427-435e-acbe-b6b090dde0d3"
Sep 07 16:20:39.897060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.897031    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeReady"
Sep 07 16:20:39.897498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.897160    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 16:20:39.939608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.939575    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mgfc2"]
Sep 07 16:20:39.974966 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.974932    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5rwnq"]
Sep 07 16:20:39.974966 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.974957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:39.977440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.977416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 16:20:39.977622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.977447    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c2wmp\""
Sep 07 16:20:39.977622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.977452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 16:20:39.997176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.997153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgfc2"]
Sep 07 16:20:39.997277 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.997182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5rwnq"]
Sep 07 16:20:39.997277 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.997248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:39.999727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.999683    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 16:20:39.999933 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:39.999920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jt2mt\""
Sep 07 16:20:40.000025 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.000011    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 16:20:40.000089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.000046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 16:20:40.042736 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.042701    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v8n78"]
Sep 07 16:20:40.046913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.046889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:40.061198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.061175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v8n78"]
Sep 07 16:20:40.061312 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.061290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.064006 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.063989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 16:20:40.064123 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.064075    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 16:20:40.064210 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.064193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 16:20:40.064418 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.064402    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 16:20:40.064528 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.064513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s7rhg\""
Sep 07 16:20:40.101935 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.101903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:40.105947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.105924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa6fd336-0b62-4b47-aa45-69b561465ad3-tmp-dir\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.106065 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.105961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24c58aca-c978-4eba-a4a0-6645b48e2f38-cert\") pod \"ingress-canary-5rwnq\" (UID: \"24c58aca-c978-4eba-a4a0-6645b48e2f38\") " pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.106065 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.105983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6fd336-0b62-4b47-aa45-69b561465ad3-config-volume\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.106065 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.106029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbpc\" (UniqueName: \"kubernetes.io/projected/24c58aca-c978-4eba-a4a0-6645b48e2f38-kube-api-access-xhbpc\") pod \"ingress-canary-5rwnq\" (UID: \"24c58aca-c978-4eba-a4a0-6645b48e2f38\") " pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.106211 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.106065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crl9v\" (UniqueName: \"kubernetes.io/projected/fa6fd336-0b62-4b47-aa45-69b561465ad3-kube-api-access-crl9v\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.106211 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.106173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa6fd336-0b62-4b47-aa45-69b561465ad3-metrics-tls\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.207295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmmw\" (UniqueName: \"kubernetes.io/projected/4e4be2ed-750a-4042-a911-f20c3bb88f0b-kube-api-access-cgmmw\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.207295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa6fd336-0b62-4b47-aa45-69b561465ad3-tmp-dir\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.207295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4be2ed-750a-4042-a911-f20c3bb88f0b-crio-socket\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.207515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24c58aca-c978-4eba-a4a0-6645b48e2f38-cert\") pod \"ingress-canary-5rwnq\" (UID: \"24c58aca-c978-4eba-a4a0-6645b48e2f38\") " pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.207515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6fd336-0b62-4b47-aa45-69b561465ad3-config-volume\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.207515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbpc\" (UniqueName: \"kubernetes.io/projected/24c58aca-c978-4eba-a4a0-6645b48e2f38-kube-api-access-xhbpc\") pod \"ingress-canary-5rwnq\" (UID: \"24c58aca-c978-4eba-a4a0-6645b48e2f38\") " pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.207515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crl9v\" (UniqueName: \"kubernetes.io/projected/fa6fd336-0b62-4b47-aa45-69b561465ad3-kube-api-access-crl9v\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.207515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4be2ed-750a-4042-a911-f20c3bb88f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.207746 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4be2ed-750a-4042-a911-f20c3bb88f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.207746 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4be2ed-750a-4042-a911-f20c3bb88f0b-data-volume\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.207746 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa6fd336-0b62-4b47-aa45-69b561465ad3-metrics-tls\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.207746 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa6fd336-0b62-4b47-aa45-69b561465ad3-tmp-dir\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.207974 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.207900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6fd336-0b62-4b47-aa45-69b561465ad3-config-volume\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.211334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.211311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa6fd336-0b62-4b47-aa45-69b561465ad3-metrics-tls\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.211456 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.211440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24c58aca-c978-4eba-a4a0-6645b48e2f38-cert\") pod \"ingress-canary-5rwnq\" (UID: \"24c58aca-c978-4eba-a4a0-6645b48e2f38\") " pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.215456 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.215433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crl9v\" (UniqueName: \"kubernetes.io/projected/fa6fd336-0b62-4b47-aa45-69b561465ad3-kube-api-access-crl9v\") pod \"dns-default-mgfc2\" (UID: \"fa6fd336-0b62-4b47-aa45-69b561465ad3\") " pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.215558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.215500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbpc\" (UniqueName: \"kubernetes.io/projected/24c58aca-c978-4eba-a4a0-6645b48e2f38-kube-api-access-xhbpc\") pod \"ingress-canary-5rwnq\" (UID: \"24c58aca-c978-4eba-a4a0-6645b48e2f38\") " pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.284299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.284273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:40.304409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.304384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5rwnq"
Sep 07 16:20:40.308681 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.308662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4be2ed-750a-4042-a911-f20c3bb88f0b-data-volume\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.308784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.308694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmmw\" (UniqueName: \"kubernetes.io/projected/4e4be2ed-750a-4042-a911-f20c3bb88f0b-kube-api-access-cgmmw\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.308784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.308714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4be2ed-750a-4042-a911-f20c3bb88f0b-crio-socket\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.308784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.308741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4be2ed-750a-4042-a911-f20c3bb88f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.308922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.308801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4be2ed-750a-4042-a911-f20c3bb88f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.309009 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.308988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4be2ed-750a-4042-a911-f20c3bb88f0b-crio-socket\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.309084 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.309069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4be2ed-750a-4042-a911-f20c3bb88f0b-data-volume\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.309361 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.309344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4be2ed-750a-4042-a911-f20c3bb88f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.311508 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.311482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4be2ed-750a-4042-a911-f20c3bb88f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.317101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.317082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmmw\" (UniqueName: \"kubernetes.io/projected/4e4be2ed-750a-4042-a911-f20c3bb88f0b-kube-api-access-cgmmw\") pod \"insights-runtime-extractor-v8n78\" (UID: \"4e4be2ed-750a-4042-a911-f20c3bb88f0b\") " pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.369536 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.369474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8n78"
Sep 07 16:20:40.561612 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.561530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:20:40.561819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.561530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:20:40.561819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.561532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:20:40.565669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.565644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4f7n\""
Sep 07 16:20:40.565828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.565644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lqj4\""
Sep 07 16:20:40.565828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.565692    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 16:20:40.565828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.565644    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 16:20:40.565828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.565649    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 16:20:40.566058 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.566045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 16:20:40.604793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.604163    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgfc2"]
Sep 07 16:20:40.609509 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:40.609482    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6fd336_0b62_4b47_aa45_69b561465ad3.slice/crio-c5713218f6e5ca86d401c1980ff990e09b46021c0c469c3b3fc0c9fc6196ef08 WatchSource:0}: Error finding container c5713218f6e5ca86d401c1980ff990e09b46021c0c469c3b3fc0c9fc6196ef08: Status 404 returned error can't find the container with id c5713218f6e5ca86d401c1980ff990e09b46021c0c469c3b3fc0c9fc6196ef08
Sep 07 16:20:40.617217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.617188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5rwnq"]
Sep 07 16:20:40.620370 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:40.620112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c58aca_c978_4eba_a4a0_6645b48e2f38.slice/crio-75c04d397d51d11cbec88f19263f2d25ee2e2de072f05734c5594019c0e6f022 WatchSource:0}: Error finding container 75c04d397d51d11cbec88f19263f2d25ee2e2de072f05734c5594019c0e6f022: Status 404 returned error can't find the container with id 75c04d397d51d11cbec88f19263f2d25ee2e2de072f05734c5594019c0e6f022
Sep 07 16:20:40.672996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.672966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v8n78"]
Sep 07 16:20:40.675711 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:40.675683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4be2ed_750a_4042_a911_f20c3bb88f0b.slice/crio-7281a4e106c774ecb419dac720dfbc73afb3814b195f3baee85580ecf111ef8f WatchSource:0}: Error finding container 7281a4e106c774ecb419dac720dfbc73afb3814b195f3baee85580ecf111ef8f: Status 404 returned error can't find the container with id 7281a4e106c774ecb419dac720dfbc73afb3814b195f3baee85580ecf111ef8f
Sep 07 16:20:40.789548 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.789507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8n78" event={"ID":"4e4be2ed-750a-4042-a911-f20c3bb88f0b","Type":"ContainerStarted","Data":"7281a4e106c774ecb419dac720dfbc73afb3814b195f3baee85580ecf111ef8f"}
Sep 07 16:20:40.790795 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.790773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5rwnq" event={"ID":"24c58aca-c978-4eba-a4a0-6645b48e2f38","Type":"ContainerStarted","Data":"75c04d397d51d11cbec88f19263f2d25ee2e2de072f05734c5594019c0e6f022"}
Sep 07 16:20:40.791848 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:40.791823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgfc2" event={"ID":"fa6fd336-0b62-4b47-aa45-69b561465ad3","Type":"ContainerStarted","Data":"c5713218f6e5ca86d401c1980ff990e09b46021c0c469c3b3fc0c9fc6196ef08"}
Sep 07 16:20:41.796794 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.796732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8n78" event={"ID":"4e4be2ed-750a-4042-a911-f20c3bb88f0b","Type":"ContainerStarted","Data":"0c02eb6b1d3138455e7154af28a26e49db56b42156b17a3c36884d7254815285"}
Sep 07 16:20:41.927693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.927655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:41.961551 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.961523    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"]
Sep 07 16:20:41.984646 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.984623    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"]
Sep 07 16:20:41.984770 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.984742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:41.987202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.987181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 07 16:20:41.987323 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.987276    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 16:20:41.987393 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.987366    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 16:20:41.987445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.987402    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qq485\""
Sep 07 16:20:41.988889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.988350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 07 16:20:41.988889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:41.988366    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 16:20:42.008692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.008663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:42.094443 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.094352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:42.122510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.122423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/270b6c67-86a2-4e5b-8322-6a1a10072ca9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.122510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.122476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/270b6c67-86a2-4e5b-8322-6a1a10072ca9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.122510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.122518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270b6c67-86a2-4e5b-8322-6a1a10072ca9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.122831 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.122595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hzg\" (UniqueName: \"kubernetes.io/projected/270b6c67-86a2-4e5b-8322-6a1a10072ca9-kube-api-access-z6hzg\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.223341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.223312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hzg\" (UniqueName: \"kubernetes.io/projected/270b6c67-86a2-4e5b-8322-6a1a10072ca9-kube-api-access-z6hzg\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.223502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.223378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/270b6c67-86a2-4e5b-8322-6a1a10072ca9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.223569 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.223537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/270b6c67-86a2-4e5b-8322-6a1a10072ca9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.223602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.223584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270b6c67-86a2-4e5b-8322-6a1a10072ca9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.224535 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.224267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270b6c67-86a2-4e5b-8322-6a1a10072ca9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.227686 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.227665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/270b6c67-86a2-4e5b-8322-6a1a10072ca9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.228368 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.228345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/270b6c67-86a2-4e5b-8322-6a1a10072ca9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.233697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.233666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hzg\" (UniqueName: \"kubernetes.io/projected/270b6c67-86a2-4e5b-8322-6a1a10072ca9-kube-api-access-z6hzg\") pod \"prometheus-operator-6f4946db56-qh8fx\" (UID: \"270b6c67-86a2-4e5b-8322-6a1a10072ca9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.295136 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.295102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"
Sep 07 16:20:42.928016 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:42.927979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:43.028772 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:43.028730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:43.307940 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:43.307874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:43.410035 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:43.409979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:43.804923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:43.804895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8n78" event={"ID":"4e4be2ed-750a-4042-a911-f20c3bb88f0b","Type":"ContainerStarted","Data":"87738a17a901c64fd4c5884c1ef21d436048aa95b2ddbd88355726287245ac36"}
Sep 07 16:20:43.984256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:43.984010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-qh8fx"]
Sep 07 16:20:43.997852 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:43.997825    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270b6c67_86a2_4e5b_8322_6a1a10072ca9.slice/crio-f9229a39579a8e3a274ab3e7cc31713173afb00ebcc8d12d78a87bdb9e117af0 WatchSource:0}: Error finding container f9229a39579a8e3a274ab3e7cc31713173afb00ebcc8d12d78a87bdb9e117af0: Status 404 returned error can't find the container with id f9229a39579a8e3a274ab3e7cc31713173afb00ebcc8d12d78a87bdb9e117af0
Sep 07 16:20:44.132793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.132746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:44.233135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.233052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:44.809921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.809471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5rwnq" event={"ID":"24c58aca-c978-4eba-a4a0-6645b48e2f38","Type":"ContainerStarted","Data":"90f88c12b628aba3e7b5f2e766c80b70b25725cd61934f946864ec9f88e7edb9"}
Sep 07 16:20:44.810922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.810885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx" event={"ID":"270b6c67-86a2-4e5b-8322-6a1a10072ca9","Type":"ContainerStarted","Data":"f9229a39579a8e3a274ab3e7cc31713173afb00ebcc8d12d78a87bdb9e117af0"}
Sep 07 16:20:44.812822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.812798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgfc2" event={"ID":"fa6fd336-0b62-4b47-aa45-69b561465ad3","Type":"ContainerStarted","Data":"57f5b60a054d4a2dd0f5fcfe60ec093fa4bfcca6d3600466048964443e6862c7"}
Sep 07 16:20:44.812912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.812846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgfc2" event={"ID":"fa6fd336-0b62-4b47-aa45-69b561465ad3","Type":"ContainerStarted","Data":"1536b8dfd9b577c64d0d0520b763b385ebf6e830a167fa21b7a3f1cf9923d05e"}
Sep 07 16:20:44.812977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.812962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:44.826779 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.826712    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5rwnq" podStartSLOduration=2.725221419 podStartE2EDuration="5.826691072s" totalImagesPullingTime="3.101469653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:44.825770539 +0000 UTC m=+49.707307500" watchObservedRunningTime="2026-09-07 16:20:44.826691072 +0000 UTC m=+49.708228018"
Sep 07 16:20:44.841729 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.841684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:44.852865 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.846566    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mgfc2" podStartSLOduration=2.738778375 podStartE2EDuration="5.846546331s" totalImagesPullingTime="3.107767956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:44.846110945 +0000 UTC m=+49.727647939" watchObservedRunningTime="2026-09-07 16:20:44.846546331 +0000 UTC m=+49.728083301"
Sep 07 16:20:44.926228 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.926198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:44.929144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.929115    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b949b5cc-b8bh8"]
Sep 07 16:20:44.946353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.945232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b949b5cc-b8bh8"]
Sep 07 16:20:44.946353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.945351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:44.948416 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.948393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j7l4m\""
Sep 07 16:20:44.948557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.948422    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 16:20:44.948557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.948432    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 16:20:44.952538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.950821    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 16:20:44.952538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.951492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 16:20:44.952538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.951538    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 16:20:44.952538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.952353    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 16:20:44.952890 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.952793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 16:20:44.955231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.955210    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 16:20:44.983946 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:44.983911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:45.041179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-trusted-ca-bundle\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.041964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-service-ca\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.041964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-serving-cert\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.041964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgh2j\" (UniqueName: \"kubernetes.io/projected/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-kube-api-access-qgh2j\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.041964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-config\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.041964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-oauth-config\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.041964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.041412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-oauth-serving-cert\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.142625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-oauth-config\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.142832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-oauth-serving-cert\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.142832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-trusted-ca-bundle\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.142832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-service-ca\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.142832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-serving-cert\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.142832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgh2j\" (UniqueName: \"kubernetes.io/projected/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-kube-api-access-qgh2j\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.143104 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.142894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-config\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.144085 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.144060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-config\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.144202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.144083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-service-ca\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.144202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.144143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-trusted-ca-bundle\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.144202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.144183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-oauth-serving-cert\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.146430 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.146402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-oauth-config\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.148266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.148246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-serving-cert\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.151368 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.151345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgh2j\" (UniqueName: \"kubernetes.io/projected/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-kube-api-access-qgh2j\") pod \"console-8b949b5cc-b8bh8\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") " pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.263434 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.263395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:45.818995 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.818771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8n78" event={"ID":"4e4be2ed-750a-4042-a911-f20c3bb88f0b","Type":"ContainerStarted","Data":"f6ab5ce682c1a172e30b3d3f1b0e74bec6f41268900a09d39159b8863ce3a609"}
Sep 07 16:20:45.820706 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.820676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx" event={"ID":"270b6c67-86a2-4e5b-8322-6a1a10072ca9","Type":"ContainerStarted","Data":"3ed3137fc4aa633d09c5b15df8030e415aa741c25bb94f4790cc0d5db0db62dd"}
Sep 07 16:20:45.820871 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.820733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx" event={"ID":"270b6c67-86a2-4e5b-8322-6a1a10072ca9","Type":"ContainerStarted","Data":"558fc2530e834f47d7bc5e2405eafc838e22c02365ccd16b0e07c3e48e6aa68b"}
Sep 07 16:20:45.839527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.839472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v8n78" podStartSLOduration=2.846447062 podStartE2EDuration="5.83945693s" totalImagesPullingTime="2.993009868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:20:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:45.837892723 +0000 UTC m=+50.719429693" watchObservedRunningTime="2026-09-07 16:20:45.83945693 +0000 UTC m=+50.720993876"
Sep 07 16:20:45.855126 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.855067    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-qh8fx" podStartSLOduration=3.200443171 podStartE2EDuration="4.855050881s" totalImagesPullingTime="1.65460771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:45.854087449 +0000 UTC m=+50.735624416" watchObservedRunningTime="2026-09-07 16:20:45.855050881 +0000 UTC m=+50.736587895"
Sep 07 16:20:45.921989 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.921882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b949b5cc-b8bh8"]
Sep 07 16:20:45.925888 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:45.925858    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe2b8a0_260a_44ad_a17f_3be0aaa17136.slice/crio-3ba5a09258510ed46580e344e8b6d7df3904e0d6ee69fe0fa6c5becf50455101 WatchSource:0}: Error finding container 3ba5a09258510ed46580e344e8b6d7df3904e0d6ee69fe0fa6c5becf50455101: Status 404 returned error can't find the container with id 3ba5a09258510ed46580e344e8b6d7df3904e0d6ee69fe0fa6c5becf50455101
Sep 07 16:20:45.930932 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.930909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:45.983840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:45.983812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:46.826352 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:46.826312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b949b5cc-b8bh8" event={"ID":"4fe2b8a0-260a-44ad-a17f-3be0aaa17136","Type":"ContainerStarted","Data":"3ba5a09258510ed46580e344e8b6d7df3904e0d6ee69fe0fa6c5becf50455101"}
Sep 07 16:20:46.925339 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:46.925302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.028006 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.027967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.149296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.149260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.248171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.248134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.308491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.308456    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"]
Sep 07 16:20:47.310774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.310725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.313452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.313418    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 07 16:20:47.313452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.313451    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5dvq6\""
Sep 07 16:20:47.314016 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.313996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 07 16:20:47.322368 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.322346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"]
Sep 07 16:20:47.323286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.323263    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"]
Sep 07 16:20:47.325961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.325571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.328234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.328059    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 07 16:20:47.328234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.328074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 07 16:20:47.328234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.328145    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 07 16:20:47.328418 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.328296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bwrcg\""
Sep 07 16:20:47.337616 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.337597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"]
Sep 07 16:20:47.338307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.338289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.344167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.344141    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mhvmn"]
Sep 07 16:20:47.348302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.348273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.350989 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.350968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 16:20:47.351186 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.351170    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 16:20:47.351273 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.351188    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 16:20:47.351273 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.351207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2glkw\""
Sep 07 16:20:47.359198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.359282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.359282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/289f2444-775c-48d4-956b-14d260dca36f-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.359282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfktc\" (UniqueName: \"kubernetes.io/projected/289f2444-775c-48d4-956b-14d260dca36f-kube-api-access-rfktc\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.359419 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-root\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wn7l\" (UniqueName: \"kubernetes.io/projected/e22673d7-fc8d-4890-8251-e32f7d7f7de1-kube-api-access-5wn7l\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359576 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb611977-d6ac-4192-8c34-bdcd6fa8b612-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.359633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/289f2444-775c-48d4-956b-14d260dca36f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.359681 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-textfile\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359681 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-sys\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359787 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-tls\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359787 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cb611977-d6ac-4192-8c34-bdcd6fa8b612-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.359859 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359859 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.359927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e22673d7-fc8d-4890-8251-e32f7d7f7de1-metrics-client-ca\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.359927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj56d\" (UniqueName: \"kubernetes.io/projected/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-api-access-mj56d\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.359927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/289f2444-775c-48d4-956b-14d260dca36f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.360034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.359920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-wtmp\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.407664 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.407574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.460452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cb611977-d6ac-4192-8c34-bdcd6fa8b612-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e22673d7-fc8d-4890-8251-e32f7d7f7de1-metrics-client-ca\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj56d\" (UniqueName: \"kubernetes.io/projected/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-api-access-mj56d\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/289f2444-775c-48d4-956b-14d260dca36f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-wtmp\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.460628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/289f2444-775c-48d4-956b-14d260dca36f-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfktc\" (UniqueName: \"kubernetes.io/projected/289f2444-775c-48d4-956b-14d260dca36f-kube-api-access-rfktc\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-root\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wn7l\" (UniqueName: \"kubernetes.io/projected/e22673d7-fc8d-4890-8251-e32f7d7f7de1-kube-api-access-5wn7l\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb611977-d6ac-4192-8c34-bdcd6fa8b612-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/289f2444-775c-48d4-956b-14d260dca36f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cb611977-d6ac-4192-8c34-bdcd6fa8b612-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-textfile\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-sys\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.460914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-tls\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461344 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.461122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-textfile\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.461469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-root\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.461483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-sys\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.461994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.461623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-wtmp\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.462211 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.462186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/289f2444-775c-48d4-956b-14d260dca36f-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.462386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.462364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.462944 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:47.462602    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 07 16:20:47.462944 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:47.462666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-tls podName:cb611977-d6ac-4192-8c34-bdcd6fa8b612 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:47.962646034 +0000 UTC m=+52.844182980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-x5vz2" (UID: "cb611977-d6ac-4192-8c34-bdcd6fa8b612") : secret "kube-state-metrics-tls" not found
Sep 07 16:20:47.462944 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.462857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e22673d7-fc8d-4890-8251-e32f7d7f7de1-metrics-client-ca\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.463146 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.463079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.463590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.463539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb611977-d6ac-4192-8c34-bdcd6fa8b612-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.465649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.465628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-tls\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.465735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.465629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/289f2444-775c-48d4-956b-14d260dca36f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.465735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.465692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e22673d7-fc8d-4890-8251-e32f7d7f7de1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.465735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.465703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.465878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.465777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/289f2444-775c-48d4-956b-14d260dca36f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.473587 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.473562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wn7l\" (UniqueName: \"kubernetes.io/projected/e22673d7-fc8d-4890-8251-e32f7d7f7de1-kube-api-access-5wn7l\") pod \"node-exporter-mhvmn\" (UID: \"e22673d7-fc8d-4890-8251-e32f7d7f7de1\") " pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.473889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.473869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj56d\" (UniqueName: \"kubernetes.io/projected/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-api-access-mj56d\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.474383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.474365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfktc\" (UniqueName: \"kubernetes.io/projected/289f2444-775c-48d4-956b-14d260dca36f-kube-api-access-rfktc\") pod \"openshift-state-metrics-657fd6c54-j8vcm\" (UID: \"289f2444-775c-48d4-956b-14d260dca36f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.508310 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.508279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.607655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.607620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.622359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.622332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"
Sep 07 16:20:47.657420 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.657383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhvmn"
Sep 07 16:20:47.710459 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.709269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.808618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.808573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:47.965399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.965313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.968209 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.968185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb611977-d6ac-4192-8c34-bdcd6fa8b612-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-x5vz2\" (UID: \"cb611977-d6ac-4192-8c34-bdcd6fa8b612\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:47.998787 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:47.998737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:48.236802 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.236708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"
Sep 07 16:20:48.285911 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.285857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:48.354699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.354661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:48.360495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.360470    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:20:48.364512 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.364490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.366951 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.366867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 16:20:48.367053 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.366951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 16:20:48.367122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 16:20:48.367179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 16:20:48.367179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367157    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 16:20:48.367509 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 16:20:48.367509 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 16:20:48.367509 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367410    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 16:20:48.367509 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367451    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 16:20:48.367509 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.367489    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wm64p\""
Sep 07 16:20:48.380056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.380033    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:20:48.470288 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-web-config\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpxn\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-kube-api-access-glpxn\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-out\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470890 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.470890 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.470839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.503472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.503394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:48.571378 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.571466 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.571466 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.571565 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-web-config\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.571565 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpxn\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-kube-api-access-glpxn\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.571999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-out\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.572379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.573745 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.572879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.573745 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.573166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.573745 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.573706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.575026 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.574982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.576933 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.576885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-out\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.577466 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.577401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.577651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.577627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.578014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.577971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.578615 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.578547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-web-config\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.579083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.579053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.579083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.579058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.581245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.581220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.581463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.581445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpxn\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-kube-api-access-glpxn\") pod \"alertmanager-main-0\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.591851 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:48.591814    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22673d7_fc8d_4890_8251_e32f7d7f7de1.slice/crio-50dbd83aa06bd11c3255a388235649def52974d9d4adfd465c0b7598a0b50ec4 WatchSource:0}: Error finding container 50dbd83aa06bd11c3255a388235649def52974d9d4adfd465c0b7598a0b50ec4: Status 404 returned error can't find the container with id 50dbd83aa06bd11c3255a388235649def52974d9d4adfd465c0b7598a0b50ec4
Sep 07 16:20:48.597482 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.597393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:48.676248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.676219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:20:48.836056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.835846    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm"]
Sep 07 16:20:48.840798 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.839737    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2"]
Sep 07 16:20:48.843666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.843639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b949b5cc-b8bh8" event={"ID":"4fe2b8a0-260a-44ad-a17f-3be0aaa17136","Type":"ContainerStarted","Data":"2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd"}
Sep 07 16:20:48.845280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.845254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhvmn" event={"ID":"e22673d7-fc8d-4890-8251-e32f7d7f7de1","Type":"ContainerStarted","Data":"50dbd83aa06bd11c3255a388235649def52974d9d4adfd465c0b7598a0b50ec4"}
Sep 07 16:20:48.845502 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:48.845472    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb611977_d6ac_4192_8c34_bdcd6fa8b612.slice/crio-cd3203a01fd8f2a9d137d5a124f73655212b2ddf38fa0458d07a67f41d68ff34 WatchSource:0}: Error finding container cd3203a01fd8f2a9d137d5a124f73655212b2ddf38fa0458d07a67f41d68ff34: Status 404 returned error can't find the container with id cd3203a01fd8f2a9d137d5a124f73655212b2ddf38fa0458d07a67f41d68ff34
Sep 07 16:20:48.860296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.860257    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8b949b5cc-b8bh8" podStartSLOduration=2.181004206 podStartE2EDuration="4.860243293s" totalImagesPullingTime="2.679239087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:48.859009303 +0000 UTC m=+53.740546273" watchObservedRunningTime="2026-09-07 16:20:48.860243293 +0000 UTC m=+53.741780257"
Sep 07 16:20:48.957024 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:48.956998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:20:48.960113 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:48.960089    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d500984_35dc_4458_b41b_a6c2fcb3cf0f.slice/crio-4252989be79d28e5401264b65f5444ec97f1a4c98fccd4e8adc78d4fd46b8d75 WatchSource:0}: Error finding container 4252989be79d28e5401264b65f5444ec97f1a4c98fccd4e8adc78d4fd46b8d75: Status 404 returned error can't find the container with id 4252989be79d28e5401264b65f5444ec97f1a4c98fccd4e8adc78d4fd46b8d75
Sep 07 16:20:49.735376 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.735291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:49.838876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.838835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:49.850215 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.850176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"4252989be79d28e5401264b65f5444ec97f1a4c98fccd4e8adc78d4fd46b8d75"}
Sep 07 16:20:49.851722 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.851659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2" event={"ID":"cb611977-d6ac-4192-8c34-bdcd6fa8b612","Type":"ContainerStarted","Data":"cd3203a01fd8f2a9d137d5a124f73655212b2ddf38fa0458d07a67f41d68ff34"}
Sep 07 16:20:49.853558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.853531    2577 generic.go:356] "Generic (PLEG): container finished" podID="e22673d7-fc8d-4890-8251-e32f7d7f7de1" containerID="9914def87830a0badefac10695b9f9118ba786ca8a8972c2d13b307660d9f073" exitCode=0
Sep 07 16:20:49.853674 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.853640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhvmn" event={"ID":"e22673d7-fc8d-4890-8251-e32f7d7f7de1","Type":"ContainerDied","Data":"9914def87830a0badefac10695b9f9118ba786ca8a8972c2d13b307660d9f073"}
Sep 07 16:20:49.855655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.855634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm" event={"ID":"289f2444-775c-48d4-956b-14d260dca36f","Type":"ContainerStarted","Data":"97e68e10a44dad22864e7d725e5b9404f6d08e34191103d7befe3fea28b938ee"}
Sep 07 16:20:49.855727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.855688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm" event={"ID":"289f2444-775c-48d4-956b-14d260dca36f","Type":"ContainerStarted","Data":"f237fc1970960d81f24a78fb35272bcae9f0428acdc55f86e6dd7daed25bcb28"}
Sep 07 16:20:49.855727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.855717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm" event={"ID":"289f2444-775c-48d4-956b-14d260dca36f","Type":"ContainerStarted","Data":"c574b925438769b67d90dac1679cdc43b6398e1a469569c1d434aae3e2253307"}
Sep 07 16:20:49.942625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:49.942595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:50.036507 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.036414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:50.259005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.258977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:50.369550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.369470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:50.459805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.459777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:50.560296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.560217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:50.861331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.861295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhvmn" event={"ID":"e22673d7-fc8d-4890-8251-e32f7d7f7de1","Type":"ContainerStarted","Data":"35fd6daeaff27d594b6bd0f7a36425e1345965d8d5c95930e66c3062164e679e"}
Sep 07 16:20:50.861806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.861357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhvmn" event={"ID":"e22673d7-fc8d-4890-8251-e32f7d7f7de1","Type":"ContainerStarted","Data":"c60a4df4294d21ac65b00b177d83a28c5983c9ebb1862b442c6e813cd9185e7e"}
Sep 07 16:20:50.863843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.863805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm" event={"ID":"289f2444-775c-48d4-956b-14d260dca36f","Type":"ContainerStarted","Data":"558ecb23ce43c21dc13361a5bf4d834148d68a50d365a9da46c712a168d99604"}
Sep 07 16:20:50.865517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.865481    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a" exitCode=0
Sep 07 16:20:50.865630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.865562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"}
Sep 07 16:20:50.874261 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.874232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2" event={"ID":"cb611977-d6ac-4192-8c34-bdcd6fa8b612","Type":"ContainerStarted","Data":"d625cef3a027304df1037498699a5a34aded43a3c820553a4ebbdc5c1d792f01"}
Sep 07 16:20:50.874366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.874313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2" event={"ID":"cb611977-d6ac-4192-8c34-bdcd6fa8b612","Type":"ContainerStarted","Data":"3dbba2ff01d7c6dcb3a862713c9b63550ed482a6bfb0493da04af7e9a55ac152"}
Sep 07 16:20:50.874366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.874334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2" event={"ID":"cb611977-d6ac-4192-8c34-bdcd6fa8b612","Type":"ContainerStarted","Data":"f78fc744efb2f689eb98e11e2a67be5d96810a681327f433ade252b5aca5e485"}
Sep 07 16:20:50.882919 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.882867    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mhvmn" podStartSLOduration=2.97442222 podStartE2EDuration="3.882853543s" totalImagesPullingTime="824.161605ms" totalInitContainerRuntime="84.269718ms" isStatefulPod=true podCreationTimestamp="2026-09-07 16:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:50.881134642 +0000 UTC m=+55.762671606" watchObservedRunningTime="2026-09-07 16:20:50.882853543 +0000 UTC m=+55.764390505"
Sep 07 16:20:50.896963 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.896912    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-j8vcm" podStartSLOduration=2.344521389 podStartE2EDuration="3.896896469s" totalImagesPullingTime="1.55237508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:50.896415278 +0000 UTC m=+55.777952258" watchObservedRunningTime="2026-09-07 16:20:50.896896469 +0000 UTC m=+55.778433436"
Sep 07 16:20:50.942621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:50.942495    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-x5vz2" podStartSLOduration=2.293402718 podStartE2EDuration="3.942471303s" totalImagesPullingTime="1.649068585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:50.941978977 +0000 UTC m=+55.823515946" watchObservedRunningTime="2026-09-07 16:20:50.942471303 +0000 UTC m=+55.824008271"
Sep 07 16:20:51.283893 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.283821    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"]
Sep 07 16:20:51.300070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.300048    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"]
Sep 07 16:20:51.300200 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.300187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.302766 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.302733    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 07 16:20:51.302884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.302825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 07 16:20:51.302884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.302859    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 07 16:20:51.302884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.302799    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5jore3rm7a5k7\""
Sep 07 16:20:51.303032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.302785    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7rg7g\""
Sep 07 16:20:51.303032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.302983    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 07 16:20:51.303182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.303170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 07 16:20:51.401683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.401644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/713b580e-0a97-45d9-9dbd-c72372688503-metrics-client-ca\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.401894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.401705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.401894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.401825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-tls\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.401894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.401868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.402060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.401978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.402060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.402007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krsfd\" (UniqueName: \"kubernetes.io/projected/713b580e-0a97-45d9-9dbd-c72372688503-kube-api-access-krsfd\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.402156 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.402057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-grpc-tls\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.402156 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.402098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.502703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.502661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-tls\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.502902 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.502725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.502902 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.502835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.502902 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.502860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krsfd\" (UniqueName: \"kubernetes.io/projected/713b580e-0a97-45d9-9dbd-c72372688503-kube-api-access-krsfd\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.503054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.502908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-grpc-tls\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.503054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.502953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.503054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.503017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/713b580e-0a97-45d9-9dbd-c72372688503-metrics-client-ca\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.503262 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.503077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.504005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.503977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/713b580e-0a97-45d9-9dbd-c72372688503-metrics-client-ca\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.506032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.506012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.506354 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.506280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.506499 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.506459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.506769 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.506730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-tls\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.506838 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.506814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.507313 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.507290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/713b580e-0a97-45d9-9dbd-c72372688503-secret-grpc-tls\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.516046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.516021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krsfd\" (UniqueName: \"kubernetes.io/projected/713b580e-0a97-45d9-9dbd-c72372688503-kube-api-access-krsfd\") pod \"thanos-querier-5564f7d89f-cwhn7\" (UID: \"713b580e-0a97-45d9-9dbd-c72372688503\") " pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.610827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.610744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:51.652849 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.652822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:51.716505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.716473    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56df6586f6-sbpsp"]
Sep 07 16:20:51.723695 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.723660    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:51.739855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.739805    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56df6586f6-sbpsp"]
Sep 07 16:20:51.740007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.739918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.742825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.742788    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 16:20:51.743263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.743035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 16:20:51.743263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.743096    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 16:20:51.743263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.743109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 16:20:51.743263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.743224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3hf3lbm1m4oi8\""
Sep 07 16:20:51.743493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.743472    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ps9hk\""
Sep 07 16:20:51.782101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.782070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:51.805808 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.805778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-secret-metrics-server-tls\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.805982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.805813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f8088f-2f88-4048-a107-715c505b8027-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.805982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.805872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-client-ca-bundle\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.805982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.805916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcml2\" (UniqueName: \"kubernetes.io/projected/76f8088f-2f88-4048-a107-715c505b8027-kube-api-access-lcml2\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.805982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.805980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-secret-metrics-server-client-certs\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.806188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.806018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76f8088f-2f88-4048-a107-715c505b8027-audit-log\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.806188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.806091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76f8088f-2f88-4048-a107-715c505b8027-metrics-server-audit-profiles\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.906855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.906825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-secret-metrics-server-client-certs\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907233 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76f8088f-2f88-4048-a107-715c505b8027-audit-log\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76f8088f-2f88-4048-a107-715c505b8027-metrics-server-audit-profiles\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907478 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-secret-metrics-server-tls\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907536 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76f8088f-2f88-4048-a107-715c505b8027-audit-log\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907536 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f8088f-2f88-4048-a107-715c505b8027-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-client-ca-bundle\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.907633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.907596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcml2\" (UniqueName: \"kubernetes.io/projected/76f8088f-2f88-4048-a107-715c505b8027-kube-api-access-lcml2\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.908133 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.908078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f8088f-2f88-4048-a107-715c505b8027-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.908744 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.908721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76f8088f-2f88-4048-a107-715c505b8027-metrics-server-audit-profiles\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.909597 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.909568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-secret-metrics-server-client-certs\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.910141 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.910123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-client-ca-bundle\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.910415 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.910398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76f8088f-2f88-4048-a107-715c505b8027-secret-metrics-server-tls\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:51.913833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.913813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"]
Sep 07 16:20:51.915961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:51.915942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcml2\" (UniqueName: \"kubernetes.io/projected/76f8088f-2f88-4048-a107-715c505b8027-kube-api-access-lcml2\") pod \"metrics-server-56df6586f6-sbpsp\" (UID: \"76f8088f-2f88-4048-a107-715c505b8027\") " pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:52.052062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.051908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:20:52.087961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.087928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:52.091200 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.091167    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"]
Sep 07 16:20:52.111856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.110600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"]
Sep 07 16:20:52.122870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.122489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.125282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.125259    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 07 16:20:52.125530 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.125516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gkrnf\""
Sep 07 16:20:52.187463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.187377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:52.212121 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.212090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ef743c7-f610-46b1-a8cd-f5696229f36b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.212286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.212152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/1ef743c7-f610-46b1-a8cd-f5696229f36b-kube-api-access-6z6jc\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.237612 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:52.237586    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713b580e_0a97_45d9_9dbd_c72372688503.slice/crio-ac6c9ed0eff68235035b7994bc9ff1f3ae9c825407ea7fdc04b027953b2b4a88 WatchSource:0}: Error finding container ac6c9ed0eff68235035b7994bc9ff1f3ae9c825407ea7fdc04b027953b2b4a88: Status 404 returned error can't find the container with id ac6c9ed0eff68235035b7994bc9ff1f3ae9c825407ea7fdc04b027953b2b4a88
Sep 07 16:20:52.312781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.312727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ef743c7-f610-46b1-a8cd-f5696229f36b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.312907 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:52.312887    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 07 16:20:52.312957 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.312902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/1ef743c7-f610-46b1-a8cd-f5696229f36b-kube-api-access-6z6jc\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.312989 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:20:52.312978    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef743c7-f610-46b1-a8cd-f5696229f36b-monitoring-plugin-cert podName:1ef743c7-f610-46b1-a8cd-f5696229f36b nodeName:}" failed. No retries permitted until 2026-09-07 16:20:52.812953464 +0000 UTC m=+57.694490414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1ef743c7-f610-46b1-a8cd-f5696229f36b-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-hjq5m" (UID: "1ef743c7-f610-46b1-a8cd-f5696229f36b") : secret "monitoring-plugin-cert" not found
Sep 07 16:20:52.321311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.321289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/1ef743c7-f610-46b1-a8cd-f5696229f36b-kube-api-access-6z6jc\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.362666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.362639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:52.506927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.506689    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"]
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.510170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.510382    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56df6586f6-sbpsp"]
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.512784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.513093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.513301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.513482    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.513634    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tg28l\""
Sep 07 16:20:52.515177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.513802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 07 16:20:52.519436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.519420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 07 16:20:52.525481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.525464    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"]
Sep 07 16:20:52.617019 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.616990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-serving-certs-ca-bundle\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-metrics-client-ca\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-secret-telemeter-client\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617268 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-federate-client-tls\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617268 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617268 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frznv\" (UniqueName: \"kubernetes.io/projected/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-kube-api-access-frznv\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.617365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.617301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-telemeter-client-tls\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-telemeter-client-tls\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-serving-certs-ca-bundle\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-metrics-client-ca\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-secret-telemeter-client\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-federate-client-tls\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.718617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.718612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frznv\" (UniqueName: \"kubernetes.io/projected/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-kube-api-access-frznv\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.719408 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.719274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-serving-certs-ca-bundle\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.719652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.719627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.719731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.719714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-metrics-client-ca\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.721207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.721171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.721299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.721188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-secret-telemeter-client\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.721299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.721238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-federate-client-tls\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.721359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.721306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-telemeter-client-tls\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.726407 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.726385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frznv\" (UniqueName: \"kubernetes.io/projected/8eb07a7c-6eb8-4b7c-841a-4f41304e446e-kube-api-access-frznv\") pod \"telemeter-client-7bfffc7ff4-nbzpp\" (UID: \"8eb07a7c-6eb8-4b7c-841a-4f41304e446e\") " pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.819242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.819207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ef743c7-f610-46b1-a8cd-f5696229f36b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.822013 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.821989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ef743c7-f610-46b1-a8cd-f5696229f36b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hjq5m\" (UID: \"1ef743c7-f610-46b1-a8cd-f5696229f36b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:52.852942 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.852901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"
Sep 07 16:20:52.884943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.884908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"}
Sep 07 16:20:52.885128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.884990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"}
Sep 07 16:20:52.885128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.885027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"}
Sep 07 16:20:52.885128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.885060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"}
Sep 07 16:20:52.885128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.885095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"}
Sep 07 16:20:52.886259 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.886231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp" event={"ID":"76f8088f-2f88-4048-a107-715c505b8027","Type":"ContainerStarted","Data":"09d218f77ba5e641ebfc0e21e1d86c8cfea37de25a1d914457ade5b360bc0f8e"}
Sep 07 16:20:52.887624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.887595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"ac6c9ed0eff68235035b7994bc9ff1f3ae9c825407ea7fdc04b027953b2b4a88"}
Sep 07 16:20:52.935596 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.935562    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b949b5cc-b8bh8"]
Sep 07 16:20:52.960919 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.960440    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcdfc968d-dhsnf"]
Sep 07 16:20:52.963270 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.963244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:52.973090 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:52.973068    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcdfc968d-dhsnf"]
Sep 07 16:20:53.021919 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.021884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-trusted-ca-bundle\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.022090 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.021954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-service-ca\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.022090 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.022003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxrn\" (UniqueName: \"kubernetes.io/projected/57037942-239a-4f9a-aee9-93fd3143dd49-kube-api-access-7hxrn\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.022090 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.022042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-oauth-serving-cert\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.022250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.022095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-serving-cert\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.022250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.022134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-console-config\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.022250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.022169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-oauth-config\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.033239 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.032957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:53.123409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-trusted-ca-bundle\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.123557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-service-ca\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.123557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxrn\" (UniqueName: \"kubernetes.io/projected/57037942-239a-4f9a-aee9-93fd3143dd49-kube-api-access-7hxrn\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.123557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-oauth-serving-cert\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.123557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-serving-cert\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.123774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-console-config\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.123774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.123625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-oauth-config\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.127550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.127515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-trusted-ca-bundle\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.128190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.128151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-service-ca\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.128303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.128189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-console-config\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.128676 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.128616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-oauth-config\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.129017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.128960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-oauth-serving-cert\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.134028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.134003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-serving-cert\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.135282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.135203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp"]
Sep 07 16:20:53.135376 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.135287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxrn\" (UniqueName: \"kubernetes.io/projected/57037942-239a-4f9a-aee9-93fd3143dd49-kube-api-access-7hxrn\") pod \"console-5bcdfc968d-dhsnf\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") " pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.138827 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:53.138804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb07a7c_6eb8_4b7c_841a_4f41304e446e.slice/crio-5916da55515920d43353e9cd44b688fe02dd3bf780e31174233e16a53bec46cc WatchSource:0}: Error finding container 5916da55515920d43353e9cd44b688fe02dd3bf780e31174233e16a53bec46cc: Status 404 returned error can't find the container with id 5916da55515920d43353e9cd44b688fe02dd3bf780e31174233e16a53bec46cc
Sep 07 16:20:53.277100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.277002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:20:53.320288 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.320239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"]
Sep 07 16:20:53.400120 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:53.400081    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef743c7_f610_46b1_a8cd_f5696229f36b.slice/crio-134631fa78698c9e28f3218aadc0f9052d13d47aa798729b955d8d0206a1d95e WatchSource:0}: Error finding container 134631fa78698c9e28f3218aadc0f9052d13d47aa798729b955d8d0206a1d95e: Status 404 returned error can't find the container with id 134631fa78698c9e28f3218aadc0f9052d13d47aa798729b955d8d0206a1d95e
Sep 07 16:20:53.585615 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.585572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:53.586177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.585854    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:20:53.592160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.592141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.596997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-69jbg0nev7r97\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597330    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597453    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597487    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 16:20:53.597599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597550    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 16:20:53.598163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.597642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 16:20:53.599599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.598512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 16:20:53.599599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.598525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 16:20:53.599599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.598788    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 16:20:53.601952 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.600545    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r8lgc\""
Sep 07 16:20:53.601952 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.600784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 16:20:53.604412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.603421    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 16:20:53.609724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.609703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 16:20:53.610218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.610199    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-config\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4l7\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-kube-api-access-2g4l7\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.632629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.633539 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.633539 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.633539 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.633539 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.631942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.710124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.709687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcdfc968d-dhsnf"]
Sep 07 16:20:53.733238 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-config\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733238 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4l7\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-kube-api-access-2g4l7\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.733833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.733835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.735122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.735422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.735726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.735872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.736236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.737789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.738240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.738015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.745282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.741427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.745282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.742180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.745282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.745222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4l7\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-kube-api-access-2g4l7\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.748528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.748803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.748969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.748992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-config\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749530 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.749501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749680 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.749662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.749990 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.749923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.750228 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.750204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.750228 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.750216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.750547 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.750513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.895671 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.895605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m" event={"ID":"1ef743c7-f610-46b1-a8cd-f5696229f36b","Type":"ContainerStarted","Data":"134631fa78698c9e28f3218aadc0f9052d13d47aa798729b955d8d0206a1d95e"}
Sep 07 16:20:53.897018 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.896982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp" event={"ID":"8eb07a7c-6eb8-4b7c-841a-4f41304e446e","Type":"ContainerStarted","Data":"5916da55515920d43353e9cd44b688fe02dd3bf780e31174233e16a53bec46cc"}
Sep 07 16:20:53.905002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.904966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerStarted","Data":"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"}
Sep 07 16:20:53.908453 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.908425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:20:53.930306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:53.930241    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.112351029 podStartE2EDuration="5.930220255s" totalImagesPullingTime="3.801573112s" totalInitContainerRuntime="16.296114ms" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:53.929108733 +0000 UTC m=+58.810645735" watchObservedRunningTime="2026-09-07 16:20:53.930220255 +0000 UTC m=+58.811757224"
Sep 07 16:20:54.513236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.513193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:54.613977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.613938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:54.692159 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:54.692116    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57037942_239a_4f9a_aee9_93fd3143dd49.slice/crio-82b46990677283a89297e41ac0f4131eec35a00b96dd4238772cdf242fc69ab7 WatchSource:0}: Error finding container 82b46990677283a89297e41ac0f4131eec35a00b96dd4238772cdf242fc69ab7: Status 404 returned error can't find the container with id 82b46990677283a89297e41ac0f4131eec35a00b96dd4238772cdf242fc69ab7
Sep 07 16:20:54.753124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.753091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hcbml"
Sep 07 16:20:54.823412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.823332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgfc2"
Sep 07 16:20:54.865433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.865384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:54.912404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.912376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcdfc968d-dhsnf" event={"ID":"57037942-239a-4f9a-aee9-93fd3143dd49","Type":"ContainerStarted","Data":"82b46990677283a89297e41ac0f4131eec35a00b96dd4238772cdf242fc69ab7"}
Sep 07 16:20:54.942311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:54.942273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:55.263600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.263534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:20:55.381195 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.379228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:20:55.381195 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:20:55.380525    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34498079_d1c4_45ea_8522_8e32826a08c0.slice/crio-e4aec86878530491053ae80b46152518785a7050527611cefae19c377f852de3 WatchSource:0}: Error finding container e4aec86878530491053ae80b46152518785a7050527611cefae19c377f852de3: Status 404 returned error can't find the container with id e4aec86878530491053ae80b46152518785a7050527611cefae19c377f852de3
Sep 07 16:20:55.919159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.919124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp" event={"ID":"76f8088f-2f88-4048-a107-715c505b8027","Type":"ContainerStarted","Data":"6e52a31239b1613674101332da25949387ae9f37e46fb273f5ff15e63da51bce"}
Sep 07 16:20:55.926144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.926104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"a68890eaa066c74c4efe7b7157df3586f3af8e6593ee4e1feb70fbecc47dcf31"}
Sep 07 16:20:55.926307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.926179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"0279fc1d538bb7139f96702969bece042fd919267bd824ac7aa5c0641a43df57"}
Sep 07 16:20:55.926307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.926217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"90959111417ff5b499a9464282e095f3f7f1368e66ee584a81927e4403325c5b"}
Sep 07 16:20:55.926307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.926255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"de253fd3135a7b69056b426a745a8997637ad5c60432af4bebe2044a22ab922b"}
Sep 07 16:20:55.926307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.926292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"b76f70f820778fcf4a9470894ba78b94161e6e17bc0975d8a432500de4f1c7b2"}
Sep 07 16:20:55.926527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.926327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" event={"ID":"713b580e-0a97-45d9-9dbd-c72372688503","Type":"ContainerStarted","Data":"0065182b9481f288af1d3c343147c425e224e88f4b9afc157c8520882185c9c6"}
Sep 07 16:20:55.927175 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.927153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:20:55.929400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.929000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m" event={"ID":"1ef743c7-f610-46b1-a8cd-f5696229f36b","Type":"ContainerStarted","Data":"e6e2ff16b3296911ab1edb6457ac08eae6a2750aeec97231e95e1a12c30e2931"}
Sep 07 16:20:55.929400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.929045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:55.934524 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.931175    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57" exitCode=0
Sep 07 16:20:55.934524 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.931259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"}
Sep 07 16:20:55.934524 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.931325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"e4aec86878530491053ae80b46152518785a7050527611cefae19c377f852de3"}
Sep 07 16:20:55.935301 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.934986    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m"
Sep 07 16:20:55.935342 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.935294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcdfc968d-dhsnf" event={"ID":"57037942-239a-4f9a-aee9-93fd3143dd49","Type":"ContainerStarted","Data":"29a8eeda2172270d561bca07d5e8267b553ce54158d7e06ebcee7378a23bd60d"}
Sep 07 16:20:55.935690 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.935649    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp" podStartSLOduration=2.387963233 podStartE2EDuration="4.935635072s" totalImagesPullingTime="2.547671839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:55.933493483 +0000 UTC m=+60.815030452" watchObservedRunningTime="2026-09-07 16:20:55.935635072 +0000 UTC m=+60.817172121"
Sep 07 16:20:55.972473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.972411    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hjq5m" podStartSLOduration=2.223795683 podStartE2EDuration="3.972394014s" totalImagesPullingTime="1.748598331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:55.97163204 +0000 UTC m=+60.853169014" watchObservedRunningTime="2026-09-07 16:20:55.972394014 +0000 UTC m=+60.853930960"
Sep 07 16:20:55.992820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:55.992738    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7" podStartSLOduration=2.144180883 podStartE2EDuration="4.992716048s" totalImagesPullingTime="2.848535165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:55.99206965 +0000 UTC m=+60.873606641" watchObservedRunningTime="2026-09-07 16:20:55.992716048 +0000 UTC m=+60.874253018"
Sep 07 16:20:56.010493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:56.010436    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bcdfc968d-dhsnf" podStartSLOduration=4.01041853 podStartE2EDuration="4.01041853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:56.008907777 +0000 UTC m=+60.890444747" watchObservedRunningTime="2026-09-07 16:20:56.01041853 +0000 UTC m=+60.891955497"
Sep 07 16:20:56.912132 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:56.912078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:56.942414 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:56.942367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp" event={"ID":"8eb07a7c-6eb8-4b7c-841a-4f41304e446e","Type":"ContainerStarted","Data":"5730a8279b8b13d6b4b088541673c2ebee5ddba0c12c3fdf7ebfa47efe6b041f"}
Sep 07 16:20:56.942879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:56.942478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp" event={"ID":"8eb07a7c-6eb8-4b7c-841a-4f41304e446e","Type":"ContainerStarted","Data":"1909071dcd8f9e34f9c80ef46d3313f7f717d99861918efcd805938692494f91"}
Sep 07 16:20:56.942879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:56.942509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp" event={"ID":"8eb07a7c-6eb8-4b7c-841a-4f41304e446e","Type":"ContainerStarted","Data":"fc1c26503fde8f8c6fa4aaa8949b6c6c509c18e448164092c73fc908e8f93b52"}
Sep 07 16:20:56.963302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:56.963239    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bfffc7ff4-nbzpp" podStartSLOduration=1.942536362 podStartE2EDuration="4.963219014s" totalImagesPullingTime="3.020682652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:56.961964645 +0000 UTC m=+61.843501632" watchObservedRunningTime="2026-09-07 16:20:56.963219014 +0000 UTC m=+61.844755983"
Sep 07 16:20:57.013297 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:57.013232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:57.114476 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:57.114412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:57.216835 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:57.216122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:20:58.935541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:58.934781    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcdfc968d-dhsnf"]
Sep 07 16:20:59.963028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.962994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"}
Sep 07 16:20:59.963350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.963057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"}
Sep 07 16:20:59.963350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.963082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"}
Sep 07 16:20:59.963350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.963105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"}
Sep 07 16:20:59.963350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.963126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"}
Sep 07 16:20:59.963350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.963149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerStarted","Data":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"}
Sep 07 16:20:59.991172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:20:59.991070    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.447954003 podStartE2EDuration="6.991051354s" totalImagesPullingTime="3.528906791s" totalInitContainerRuntime="14.19056ms" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:59.98881897 +0000 UTC m=+64.870355939" watchObservedRunningTime="2026-09-07 16:20:59.991051354 +0000 UTC m=+64.872588321"
Sep 07 16:21:00.113082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:00.113042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:00.240875 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:00.240847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:00.741654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:00.741622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:00.841743 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:00.841706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:01.326122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.326063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:21:01.326122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.326126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:21:01.328864 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.328845    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 16:21:01.328909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.328889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 16:21:01.339385 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.339352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/044272e8-5427-435e-acbe-b6b090dde0d3-original-pull-secret\") pod \"global-pull-secret-syncer-n2rmj\" (UID: \"044272e8-5427-435e-acbe-b6b090dde0d3\") " pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:21:01.339632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.339616    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 16:21:01.350465 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.350436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2j9d\" (UniqueName: \"kubernetes.io/projected/4935ae52-b9c9-40da-9482-714b29e4048e-kube-api-access-p2j9d\") pod \"network-check-target-njkrv\" (UID: \"4935ae52-b9c9-40da-9482-714b29e4048e\") " pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:21:01.426780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.426725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:21:01.429228 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.429204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 16:21:01.439304 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.439277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f00fc3-8fcf-4f2c-b951-15e809f38527-metrics-certs\") pod \"network-metrics-daemon-9pgv4\" (UID: \"55f00fc3-8fcf-4f2c-b951-15e809f38527\") " pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:21:01.575793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.575743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4f7n\""
Sep 07 16:21:01.578703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.578647    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2rmj"
Sep 07 16:21:01.584217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.584197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pgv4"
Sep 07 16:21:01.589524 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.589505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lqj4\""
Sep 07 16:21:01.597590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.597568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:21:01.848730 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.848704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n2rmj"]
Sep 07 16:21:01.851072 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:21:01.851051    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044272e8_5427_435e_acbe_b6b090dde0d3.slice/crio-6020d3fdfb3c1fbcc3ffad6890271cf54c6819c31194699453f859a2a60986f4 WatchSource:0}: Error finding container 6020d3fdfb3c1fbcc3ffad6890271cf54c6819c31194699453f859a2a60986f4: Status 404 returned error can't find the container with id 6020d3fdfb3c1fbcc3ffad6890271cf54c6819c31194699453f859a2a60986f4
Sep 07 16:21:01.858969 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.858948    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9pgv4"]
Sep 07 16:21:01.860911 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:21:01.860890    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f00fc3_8fcf_4f2c_b951_15e809f38527.slice/crio-aa87aef0118835b52525745ab6b5bbe99705d89887dd6554a22daa217bdbe3d3 WatchSource:0}: Error finding container aa87aef0118835b52525745ab6b5bbe99705d89887dd6554a22daa217bdbe3d3: Status 404 returned error can't find the container with id aa87aef0118835b52525745ab6b5bbe99705d89887dd6554a22daa217bdbe3d3
Sep 07 16:21:01.879306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.879286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njkrv"]
Sep 07 16:21:01.881507 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:21:01.881485    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4935ae52_b9c9_40da_9482_714b29e4048e.slice/crio-b20169357d411fa04e8260a911804cdb14a56e960263915ab57e5c5ee3aec855 WatchSource:0}: Error finding container b20169357d411fa04e8260a911804cdb14a56e960263915ab57e5c5ee3aec855: Status 404 returned error can't find the container with id b20169357d411fa04e8260a911804cdb14a56e960263915ab57e5c5ee3aec855
Sep 07 16:21:01.972305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.972269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9pgv4" event={"ID":"55f00fc3-8fcf-4f2c-b951-15e809f38527","Type":"ContainerStarted","Data":"aa87aef0118835b52525745ab6b5bbe99705d89887dd6554a22daa217bdbe3d3"}
Sep 07 16:21:01.973495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.973474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njkrv" event={"ID":"4935ae52-b9c9-40da-9482-714b29e4048e","Type":"ContainerStarted","Data":"b20169357d411fa04e8260a911804cdb14a56e960263915ab57e5c5ee3aec855"}
Sep 07 16:21:01.974588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:01.974569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n2rmj" event={"ID":"044272e8-5427-435e-acbe-b6b090dde0d3","Type":"ContainerStarted","Data":"6020d3fdfb3c1fbcc3ffad6890271cf54c6819c31194699453f859a2a60986f4"}
Sep 07 16:21:02.956007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:02.955978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5564f7d89f-cwhn7"
Sep 07 16:21:03.277351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:03.277324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:21:03.909201 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:03.909162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:03.949138 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:03.949035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:03.989889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:03.989859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9pgv4" event={"ID":"55f00fc3-8fcf-4f2c-b951-15e809f38527","Type":"ContainerStarted","Data":"fd53db9d64c862c5e2c35ca8219caaa4b76f9645fd362fdbae0a797cd5bcd7db"}
Sep 07 16:21:03.990331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:03.989913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9pgv4" event={"ID":"55f00fc3-8fcf-4f2c-b951-15e809f38527","Type":"ContainerStarted","Data":"060baf50d3d6809c2d1c438a35c547def6f5eb6734fa8e5a103623363ff3af5c"}
Sep 07 16:21:04.007034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:04.006972    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9pgv4" podStartSLOduration=67.843566912 podStartE2EDuration="1m9.006951499s" totalImagesPullingTime="1.163384587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:21:04.005516605 +0000 UTC m=+68.887053573" watchObservedRunningTime="2026-09-07 16:21:04.006951499 +0000 UTC m=+68.888488468"
Sep 07 16:21:04.080648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:04.080609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:04.181045 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:04.180967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:04.231774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:04.231718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:06.166852 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:06.166815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:06.645375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:06.645342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:06.698260 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:06.698233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:07.008520 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:07.008481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njkrv" event={"ID":"4935ae52-b9c9-40da-9482-714b29e4048e","Type":"ContainerStarted","Data":"562b1f528cf7e7385b5795cb962b7d8116f905895a0a39aef7cb95b3ed85f973"}
Sep 07 16:21:07.008720 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:07.008578    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:21:07.010053 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:07.010027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n2rmj" event={"ID":"044272e8-5427-435e-acbe-b6b090dde0d3","Type":"ContainerStarted","Data":"3587cfdb0b599b587e9aa3126fba49ad109df986c4bb0fbc7a81b97d894e2d82"}
Sep 07 16:21:07.025714 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:07.025660    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njkrv" podStartSLOduration=67.453324647 podStartE2EDuration="1m12.025645469s" totalImagesPullingTime="4.572320822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:21:07.023362992 +0000 UTC m=+71.904899963" watchObservedRunningTime="2026-09-07 16:21:07.025645469 +0000 UTC m=+71.907182436"
Sep 07 16:21:07.040022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:07.039976    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n2rmj" podStartSLOduration=67.431447278 podStartE2EDuration="1m12.039961702s" totalImagesPullingTime="4.608514424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:21:07.038285526 +0000 UTC m=+71.919822495" watchObservedRunningTime="2026-09-07 16:21:07.039961702 +0000 UTC m=+71.921498649"
Sep 07 16:21:09.709973 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:09.709935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:09.810471 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:09.810435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:09.860796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:09.860761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:12.053319 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:12.053284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:21:12.053689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:12.053346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:21:17.959735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:17.959604    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8b949b5cc-b8bh8" podUID="4fe2b8a0-260a-44ad-a17f-3be0aaa17136" containerName="console" containerID="cri-o://2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd" gracePeriod=15
Sep 07 16:21:18.200202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.200166    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:21:18.285182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-service-ca\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgh2j\" (UniqueName: \"kubernetes.io/projected/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-kube-api-access-qgh2j\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-serving-cert\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285432 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-config\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285432 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-trusted-ca-bundle\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285530 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-oauth-serving-cert\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285530 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285494    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-oauth-config\") pod \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\" (UID: \"4fe2b8a0-260a-44ad-a17f-3be0aaa17136\") "
Sep 07 16:21:18.285625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285606    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-service-ca" (OuterVolumeSpecName: "service-ca") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:18.285723 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-config" (OuterVolumeSpecName: "console-config") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:18.285723 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:18.285869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:18.285920 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285883    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-oauth-serving-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:18.285920 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285904    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:18.285999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285919    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:18.285999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.285931    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-trusted-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:18.287848 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.287818    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:18.287945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.287857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-kube-api-access-qgh2j" (OuterVolumeSpecName: "kube-api-access-qgh2j") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "kube-api-access-qgh2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:21:18.287945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.287880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4fe2b8a0-260a-44ad-a17f-3be0aaa17136" (UID: "4fe2b8a0-260a-44ad-a17f-3be0aaa17136"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:18.387161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.387129    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgh2j\" (UniqueName: \"kubernetes.io/projected/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-kube-api-access-qgh2j\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:18.387161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.387155    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-serving-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:18.387161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:18.387165    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe2b8a0-260a-44ad-a17f-3be0aaa17136-console-oauth-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:19.059325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.059291    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fe2b8a0-260a-44ad-a17f-3be0aaa17136" containerID="2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd" exitCode=0
Sep 07 16:21:19.059712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.059342    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b949b5cc-b8bh8"
Sep 07 16:21:19.059712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.059384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b949b5cc-b8bh8" event={"ID":"4fe2b8a0-260a-44ad-a17f-3be0aaa17136","Type":"ContainerDied","Data":"2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd"}
Sep 07 16:21:19.059712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.059440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b949b5cc-b8bh8" event={"ID":"4fe2b8a0-260a-44ad-a17f-3be0aaa17136","Type":"ContainerDied","Data":"3ba5a09258510ed46580e344e8b6d7df3904e0d6ee69fe0fa6c5becf50455101"}
Sep 07 16:21:19.059712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.059463    2577 scope.go:117] "RemoveContainer" containerID="2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd"
Sep 07 16:21:19.068804 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.068785    2577 scope.go:117] "RemoveContainer" containerID="2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd"
Sep 07 16:21:19.069124 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:19.069104    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd\": container with ID starting with 2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd not found: ID does not exist" containerID="2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd"
Sep 07 16:21:19.069171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.069134    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd"} err="failed to get container status \"2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd\": rpc error: code = NotFound desc = could not find container \"2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd\": container with ID starting with 2d5175e7e3d002bf2b705928b6463355deda04cbb8e42c8095abea9c144e4cdd not found: ID does not exist"
Sep 07 16:21:19.078266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.078236    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b949b5cc-b8bh8"]
Sep 07 16:21:19.083767 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.083727    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b949b5cc-b8bh8"]
Sep 07 16:21:19.566227 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:19.566196    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe2b8a0-260a-44ad-a17f-3be0aaa17136" path="/var/lib/kubelet/pods/4fe2b8a0-260a-44ad-a17f-3be0aaa17136/volumes"
Sep 07 16:21:21.652296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:21.652261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:21.699402 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:21.699374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:23.958672 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:23.958628    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bcdfc968d-dhsnf" podUID="57037942-239a-4f9a-aee9-93fd3143dd49" containerName="console" containerID="cri-o://29a8eeda2172270d561bca07d5e8267b553ce54158d7e06ebcee7378a23bd60d" gracePeriod=15
Sep 07 16:21:24.081034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.081002    2577 generic.go:356] "Generic (PLEG): container finished" podID="57037942-239a-4f9a-aee9-93fd3143dd49" containerID="29a8eeda2172270d561bca07d5e8267b553ce54158d7e06ebcee7378a23bd60d" exitCode=0
Sep 07 16:21:24.081172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.081073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcdfc968d-dhsnf" event={"ID":"57037942-239a-4f9a-aee9-93fd3143dd49","Type":"ContainerDied","Data":"29a8eeda2172270d561bca07d5e8267b553ce54158d7e06ebcee7378a23bd60d"}
Sep 07 16:21:24.221135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.221057    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:21:24.337906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.337873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-serving-cert\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338063 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.337917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-oauth-serving-cert\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338063 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.337955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-trusted-ca-bundle\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338063 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.337996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hxrn\" (UniqueName: \"kubernetes.io/projected/57037942-239a-4f9a-aee9-93fd3143dd49-kube-api-access-7hxrn\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338063 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-oauth-config\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338063 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-service-ca\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338120    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-console-config\") pod \"57037942-239a-4f9a-aee9-93fd3143dd49\" (UID: \"57037942-239a-4f9a-aee9-93fd3143dd49\") "
Sep 07 16:21:24.338427 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338393    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:24.338427 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:24.338606 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-service-ca" (OuterVolumeSpecName: "service-ca") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:24.338703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.338685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-console-config" (OuterVolumeSpecName: "console-config") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:24.340341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.340319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:24.340423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.340379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57037942-239a-4f9a-aee9-93fd3143dd49-kube-api-access-7hxrn" (OuterVolumeSpecName: "kube-api-access-7hxrn") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "kube-api-access-7hxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:21:24.340423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.340391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57037942-239a-4f9a-aee9-93fd3143dd49" (UID: "57037942-239a-4f9a-aee9-93fd3143dd49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:24.439190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439151    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-console-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:24.439190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439182    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-serving-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:24.439190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439192    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-oauth-serving-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:24.439190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439200    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-trusted-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:24.439433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439209    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxrn\" (UniqueName: \"kubernetes.io/projected/57037942-239a-4f9a-aee9-93fd3143dd49-kube-api-access-7hxrn\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:24.439433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439219    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57037942-239a-4f9a-aee9-93fd3143dd49-console-oauth-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:24.439433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:24.439226    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57037942-239a-4f9a-aee9-93fd3143dd49-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:25.087645 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:25.087622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcdfc968d-dhsnf"
Sep 07 16:21:25.087645 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:25.087628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcdfc968d-dhsnf" event={"ID":"57037942-239a-4f9a-aee9-93fd3143dd49","Type":"ContainerDied","Data":"82b46990677283a89297e41ac0f4131eec35a00b96dd4238772cdf242fc69ab7"}
Sep 07 16:21:25.088154 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:25.087680    2577 scope.go:117] "RemoveContainer" containerID="29a8eeda2172270d561bca07d5e8267b553ce54158d7e06ebcee7378a23bd60d"
Sep 07 16:21:25.107323 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:25.107293    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcdfc968d-dhsnf"]
Sep 07 16:21:25.111039 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:25.111010    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bcdfc968d-dhsnf"]
Sep 07 16:21:25.570480 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:25.570448    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57037942-239a-4f9a-aee9-93fd3143dd49" path="/var/lib/kubelet/pods/57037942-239a-4f9a-aee9-93fd3143dd49/volumes"
Sep 07 16:21:32.057485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:32.057454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:21:32.061055 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:32.061036    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56df6586f6-sbpsp"
Sep 07 16:21:33.663446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:33.663397    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:33.685475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:33.685448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:33.925185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:33.925114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:38.016022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:38.015988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njkrv"
Sep 07 16:21:43.398243 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:43.398209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:43.455374 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:43.455332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:47.606266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606233    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:21:47.606692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606521    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy" containerID="cri-o://0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4" gracePeriod=120
Sep 07 16:21:47.606692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606562    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="prom-label-proxy" containerID="cri-o://46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166" gracePeriod=120
Sep 07 16:21:47.606692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606516    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="alertmanager" containerID="cri-o://0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f" gracePeriod=120
Sep 07 16:21:47.606692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606587    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-metric" containerID="cri-o://eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0" gracePeriod=120
Sep 07 16:21:47.606692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606570    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="config-reloader" containerID="cri-o://4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb" gracePeriod=120
Sep 07 16:21:47.607001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:47.606565    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-web" containerID="cri-o://6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b" gracePeriod=120
Sep 07 16:21:48.188555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188521    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166" exitCode=0
Sep 07 16:21:48.188555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188547    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4" exitCode=0
Sep 07 16:21:48.188555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188553    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb" exitCode=0
Sep 07 16:21:48.188555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188558    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f" exitCode=0
Sep 07 16:21:48.188908 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"}
Sep 07 16:21:48.188908 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"}
Sep 07 16:21:48.188908 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"}
Sep 07 16:21:48.188908 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.188780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"}
Sep 07 16:21:48.847300 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.847278    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:48.867655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.867625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:48.960713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-volume\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-web-config\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-out\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-cluster-tls-config\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-main-db\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-tls-assets\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960938    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-trusted-ca-bundle\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.960979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.960965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.961251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glpxn\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-kube-api-access-glpxn\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.961251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-metrics-client-ca\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.961251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961050    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.961251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-main-tls\") pod \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\" (UID: \"8d500984-35dc-4458-b41b-a6c2fcb3cf0f\") "
Sep 07 16:21:48.961251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:21:48.961566 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961543    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-main-db\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:48.961741 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.961716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:48.962796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.962736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:48.964626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.964583    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-volume" (OuterVolumeSpecName: "config-volume") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:48.964889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.964741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:48.964961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.964898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:21:48.965022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.964965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-out" (OuterVolumeSpecName: "config-out") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:21:48.965480 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.965444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:48.965578 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.965543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:48.965931 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.965883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:48.966117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.966099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-kube-api-access-glpxn" (OuterVolumeSpecName: "kube-api-access-glpxn") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "kube-api-access-glpxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:21:48.969510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.969481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:48.973089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.973065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:48.976318 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:48.976286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-web-config" (OuterVolumeSpecName: "web-config") pod "8d500984-35dc-4458-b41b-a6c2fcb3cf0f" (UID: "8d500984-35dc-4458-b41b-a6c2fcb3cf0f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:49.062397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062359    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-tls-assets\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062393    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062405    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062414    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpxn\" (UniqueName: \"kubernetes.io/projected/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-kube-api-access-glpxn\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062432    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-metrics-client-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062440    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062448    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-secret-alertmanager-main-tls\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062457    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062465    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-web-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062472    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-config-out\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.062617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.062479    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d500984-35dc-4458-b41b-a6c2fcb3cf0f-cluster-tls-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:49.068563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.068525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:49.195643 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195610    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0" exitCode=0
Sep 07 16:21:49.195643 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195637    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerID="6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b" exitCode=0
Sep 07 16:21:49.195884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195698    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.195884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"}
Sep 07 16:21:49.195884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"}
Sep 07 16:21:49.195884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d500984-35dc-4458-b41b-a6c2fcb3cf0f","Type":"ContainerDied","Data":"4252989be79d28e5401264b65f5444ec97f1a4c98fccd4e8adc78d4fd46b8d75"}
Sep 07 16:21:49.196001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.195921    2577 scope.go:117] "RemoveContainer" containerID="46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"
Sep 07 16:21:49.203808 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.203790    2577 scope.go:117] "RemoveContainer" containerID="eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"
Sep 07 16:21:49.210737 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.210702    2577 scope.go:117] "RemoveContainer" containerID="0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"
Sep 07 16:21:49.218046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.218023    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:21:49.218411 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.218394    2577 scope.go:117] "RemoveContainer" containerID="6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"
Sep 07 16:21:49.222672 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.222649    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:21:49.226418 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.226403    2577 scope.go:117] "RemoveContainer" containerID="4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"
Sep 07 16:21:49.233472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.233456    2577 scope.go:117] "RemoveContainer" containerID="0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"
Sep 07 16:21:49.241098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.241076    2577 scope.go:117] "RemoveContainer" containerID="0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"
Sep 07 16:21:49.244411 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244392    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:21:49.244709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244696    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="init-config-reloader"
Sep 07 16:21:49.244774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244712    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="init-config-reloader"
Sep 07 16:21:49.244774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244720    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-web"
Sep 07 16:21:49.244774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244725    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-web"
Sep 07 16:21:49.244774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244733    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe2b8a0-260a-44ad-a17f-3be0aaa17136" containerName="console"
Sep 07 16:21:49.244774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244741    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe2b8a0-260a-44ad-a17f-3be0aaa17136" containerName="console"
Sep 07 16:21:49.244774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244771    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="config-reloader"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244778    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="config-reloader"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="prom-label-proxy"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244793    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="prom-label-proxy"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244799    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-metric"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-metric"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244814    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244822    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244831    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57037942-239a-4f9a-aee9-93fd3143dd49" containerName="console"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244835    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="57037942-239a-4f9a-aee9-93fd3143dd49" containerName="console"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244848    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="alertmanager"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244852    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="alertmanager"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244906    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="57037942-239a-4f9a-aee9-93fd3143dd49" containerName="console"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244914    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe2b8a0-260a-44ad-a17f-3be0aaa17136" containerName="console"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244920    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="config-reloader"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244926    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-metric"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244931    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244937    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="alertmanager"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244942    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="prom-label-proxy"
Sep 07 16:21:49.244947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.244948    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" containerName="kube-rbac-proxy-web"
Sep 07 16:21:49.250268 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.250250    2577 scope.go:117] "RemoveContainer" containerID="46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"
Sep 07 16:21:49.250405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.250392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.250708 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.250605    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166\": container with ID starting with 46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166 not found: ID does not exist" containerID="46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"
Sep 07 16:21:49.250708 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.250643    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"} err="failed to get container status \"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166\": rpc error: code = NotFound desc = could not find container \"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166\": container with ID starting with 46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166 not found: ID does not exist"
Sep 07 16:21:49.250708 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.250668    2577 scope.go:117] "RemoveContainer" containerID="eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"
Sep 07 16:21:49.251246 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.251226    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0\": container with ID starting with eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0 not found: ID does not exist" containerID="eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"
Sep 07 16:21:49.251299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.251253    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"} err="failed to get container status \"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0\": rpc error: code = NotFound desc = could not find container \"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0\": container with ID starting with eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0 not found: ID does not exist"
Sep 07 16:21:49.251299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.251271    2577 scope.go:117] "RemoveContainer" containerID="0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"
Sep 07 16:21:49.251589 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.251571    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4\": container with ID starting with 0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4 not found: ID does not exist" containerID="0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"
Sep 07 16:21:49.251649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.251594    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"} err="failed to get container status \"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4\": rpc error: code = NotFound desc = could not find container \"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4\": container with ID starting with 0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4 not found: ID does not exist"
Sep 07 16:21:49.251649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.251611    2577 scope.go:117] "RemoveContainer" containerID="6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"
Sep 07 16:21:49.251956 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.251938    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b\": container with ID starting with 6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b not found: ID does not exist" containerID="6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"
Sep 07 16:21:49.252002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.251962    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"} err="failed to get container status \"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b\": rpc error: code = NotFound desc = could not find container \"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b\": container with ID starting with 6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b not found: ID does not exist"
Sep 07 16:21:49.252002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.251978    2577 scope.go:117] "RemoveContainer" containerID="4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"
Sep 07 16:21:49.252285 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.252272    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb\": container with ID starting with 4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb not found: ID does not exist" containerID="4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"
Sep 07 16:21:49.252335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252288    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"} err="failed to get container status \"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb\": rpc error: code = NotFound desc = could not find container \"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb\": container with ID starting with 4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb not found: ID does not exist"
Sep 07 16:21:49.252335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252310    2577 scope.go:117] "RemoveContainer" containerID="0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"
Sep 07 16:21:49.252571 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.252556    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f\": container with ID starting with 0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f not found: ID does not exist" containerID="0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"
Sep 07 16:21:49.252624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252575    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"} err="failed to get container status \"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f\": rpc error: code = NotFound desc = could not find container \"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f\": container with ID starting with 0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f not found: ID does not exist"
Sep 07 16:21:49.252624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252589    2577 scope.go:117] "RemoveContainer" containerID="0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:49.252857    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a\": container with ID starting with 0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a not found: ID does not exist" containerID="0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252888    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"} err="failed to get container status \"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a\": rpc error: code = NotFound desc = could not find container \"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a\": container with ID starting with 0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a not found: ID does not exist"
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252908    2577 scope.go:117] "RemoveContainer" containerID="46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.252931    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253018    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wm64p\""
Sep 07 16:21:49.253071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 16:21:49.253350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 16:21:49.253350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 16:21:49.253433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 16:21:49.253483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 16:21:49.253540 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 16:21:49.253670 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253653    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166"} err="failed to get container status \"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166\": rpc error: code = NotFound desc = could not find container \"46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166\": container with ID starting with 46ecca8f23234bd4c2130eee6415a84f2976f9e93862668dec62f7c93b5b3166 not found: ID does not exist"
Sep 07 16:21:49.253707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253672    2577 scope.go:117] "RemoveContainer" containerID="eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"
Sep 07 16:21:49.253973 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253952    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0"} err="failed to get container status \"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0\": rpc error: code = NotFound desc = could not find container \"eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0\": container with ID starting with eca8add8b60b48de31fb6f4a5b5c1cf673fa67bf9707ea1b8fdbef51b910ebd0 not found: ID does not exist"
Sep 07 16:21:49.254048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.253975    2577 scope.go:117] "RemoveContainer" containerID="0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"
Sep 07 16:21:49.254277 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.254258    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4"} err="failed to get container status \"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4\": rpc error: code = NotFound desc = could not find container \"0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4\": container with ID starting with 0805724258c3c74d37a802430111d8a4aeea3c563ac5f45d831e3f4ccfa5d9a4 not found: ID does not exist"
Sep 07 16:21:49.254342 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.254279    2577 scope.go:117] "RemoveContainer" containerID="6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"
Sep 07 16:21:49.254599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.254570    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b"} err="failed to get container status \"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b\": rpc error: code = NotFound desc = could not find container \"6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b\": container with ID starting with 6063f18fdce3653168667ab133de6d484a4df3948494f7b0a988bd7a0cde1c5b not found: ID does not exist"
Sep 07 16:21:49.254675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.254601    2577 scope.go:117] "RemoveContainer" containerID="4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"
Sep 07 16:21:49.254926 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.254906    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb"} err="failed to get container status \"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb\": rpc error: code = NotFound desc = could not find container \"4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb\": container with ID starting with 4b92779c2834bcf86df5db11266ae2e51d636adbb423cbb95c3196df3be7eacb not found: ID does not exist"
Sep 07 16:21:49.254991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.254927    2577 scope.go:117] "RemoveContainer" containerID="0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"
Sep 07 16:21:49.255217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.255193    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f"} err="failed to get container status \"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f\": rpc error: code = NotFound desc = could not find container \"0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f\": container with ID starting with 0a3b3777a19ea733852ab451c73a2dda4a90a6665d0151fe672c5a630400279f not found: ID does not exist"
Sep 07 16:21:49.255351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.255218    2577 scope.go:117] "RemoveContainer" containerID="0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"
Sep 07 16:21:49.255529 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.255501    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a"} err="failed to get container status \"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a\": rpc error: code = NotFound desc = could not find container \"0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a\": container with ID starting with 0cd969dcc425f020ca8a3da5d665dc744c54c98991d8ce052794aa35e2eb1b1a not found: ID does not exist"
Sep 07 16:21:49.259707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.259671    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 16:21:49.262166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.262148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:21:49.365630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-web-config\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5172ec97-144e-46c1-b07e-73f3885cb3f8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-config-volume\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5172ec97-144e-46c1-b07e-73f3885cb3f8-config-out\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.365895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwwwx\" (UniqueName: \"kubernetes.io/projected/5172ec97-144e-46c1-b07e-73f3885cb3f8-kube-api-access-zwwwx\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.366082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5172ec97-144e-46c1-b07e-73f3885cb3f8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.366082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5172ec97-144e-46c1-b07e-73f3885cb3f8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.366082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.366082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.365975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.366082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.366011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5172ec97-144e-46c1-b07e-73f3885cb3f8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467431 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-web-config\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467431 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467431 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5172ec97-144e-46c1-b07e-73f3885cb3f8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-config-volume\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5172ec97-144e-46c1-b07e-73f3885cb3f8-config-out\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwwwx\" (UniqueName: \"kubernetes.io/projected/5172ec97-144e-46c1-b07e-73f3885cb3f8-kube-api-access-zwwwx\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5172ec97-144e-46c1-b07e-73f3885cb3f8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5172ec97-144e-46c1-b07e-73f3885cb3f8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.467687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.468023 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.468023 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5172ec97-144e-46c1-b07e-73f3885cb3f8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.468023 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.468023 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.467897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.468775 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.468732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5172ec97-144e-46c1-b07e-73f3885cb3f8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.469319 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.469293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5172ec97-144e-46c1-b07e-73f3885cb3f8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.469970 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.469937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5172ec97-144e-46c1-b07e-73f3885cb3f8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.470638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.470612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.470824 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.470803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5172ec97-144e-46c1-b07e-73f3885cb3f8-config-out\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.470911 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.470821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-web-config\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.470982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.470966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.471085 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.471063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-config-volume\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.471142 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.471074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5172ec97-144e-46c1-b07e-73f3885cb3f8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.471969 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.471933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.472047 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.471990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.472230 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.472212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5172ec97-144e-46c1-b07e-73f3885cb3f8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.475656 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.475639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwwwx\" (UniqueName: \"kubernetes.io/projected/5172ec97-144e-46c1-b07e-73f3885cb3f8-kube-api-access-zwwwx\") pod \"alertmanager-main-0\" (UID: \"5172ec97-144e-46c1-b07e-73f3885cb3f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.561496 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.561457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 16:21:49.566491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.566463    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d500984-35dc-4458-b41b-a6c2fcb3cf0f" path="/var/lib/kubelet/pods/8d500984-35dc-4458-b41b-a6c2fcb3cf0f/volumes"
Sep 07 16:21:49.610349 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.610270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:49.710027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.709989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:49.837315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:49.837231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 16:21:49.841165 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:21:49.841133    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5172ec97_144e_46c1_b07e_73f3885cb3f8.slice/crio-d23e7082487877ea1861b2106e913de5453bb09042e229dac80d7fff48ac6a75 WatchSource:0}: Error finding container d23e7082487877ea1861b2106e913de5453bb09042e229dac80d7fff48ac6a75: Status 404 returned error can't find the container with id d23e7082487877ea1861b2106e913de5453bb09042e229dac80d7fff48ac6a75
Sep 07 16:21:50.202318 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:50.202283    2577 generic.go:356] "Generic (PLEG): container finished" podID="5172ec97-144e-46c1-b07e-73f3885cb3f8" containerID="720dc5ab0be58093767efb8f854c097a69b243d2d9a389f2d3cb62c25550ef68" exitCode=0
Sep 07 16:21:50.202676 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:50.202345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerDied","Data":"720dc5ab0be58093767efb8f854c097a69b243d2d9a389f2d3cb62c25550ef68"}
Sep 07 16:21:50.202676 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:50.202457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"d23e7082487877ea1861b2106e913de5453bb09042e229dac80d7fff48ac6a75"}
Sep 07 16:21:51.211074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.211036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"876aa6c9ffcd9e33b8be07e3c67361861f36e994d62217ecb4947acb2396e01b"}
Sep 07 16:21:51.211421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.211133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"a0730c82bd59eaf9b32d98e45c2f6f0ad08b3a61adb8dd2ef2ea2fd4c4abe893"}
Sep 07 16:21:51.211421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.211172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"9105b58d0d7bcae44dab5b2f063b15d73921f29e705be5fc9090bb5e0aec3ecd"}
Sep 07 16:21:51.211421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.211199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"6de38c5629f34c21a56886eb02c970870fb716573b2ab13331d60722a67f0be7"}
Sep 07 16:21:51.211421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.211226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"bba27a69168238fdae5d1ec423cce8e828273e3efaf8ccbf44d7bea11e56506c"}
Sep 07 16:21:51.211421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.211261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5172ec97-144e-46c1-b07e-73f3885cb3f8","Type":"ContainerStarted","Data":"f510e30f9668fac4b215e8f5c6536733bd3f6d3617979e1d80688fba3a1f9094"}
Sep 07 16:21:51.239837 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.239780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.225530471 podStartE2EDuration="2.23973777s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.207299ms" isStatefulPod=false podCreationTimestamp="2026-09-07 16:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:21:51.237857658 +0000 UTC m=+116.119394627" watchObservedRunningTime="2026-09-07 16:21:51.23973777 +0000 UTC m=+116.121274739"
Sep 07 16:21:51.822929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.822894    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:21:51.823195 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.823167    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="prometheus" containerID="cri-o://c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd" gracePeriod=600
Sep 07 16:21:51.823341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.823202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2" gracePeriod=600
Sep 07 16:21:51.823341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.823229    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="config-reloader" containerID="cri-o://495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146" gracePeriod=600
Sep 07 16:21:51.823341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.823256    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="thanos-sidecar" containerID="cri-o://86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19" gracePeriod=600
Sep 07 16:21:51.823341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.823202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy" containerID="cri-o://61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc" gracePeriod=600
Sep 07 16:21:51.823341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:51.823202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-web" containerID="cri-o://8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b" gracePeriod=600
Sep 07 16:21:52.065601 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.065575    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.194610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-kubelet-serving-ca-bundle\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-0\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g4l7\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-kube-api-access-2g4l7\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-db\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-config-out\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-tls-assets\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.194863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-kube-rbac-proxy\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-metrics-client-certs\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194901    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-2\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-thanos-prometheus-http-client-file\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.194967    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-web-config\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-grpc-tls\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-metrics-client-ca\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-trusted-ca-bundle\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-1\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195140    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-config\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-serving-certs-ca-bundle\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195229    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-tls\") pod \"34498079-d1c4-45ea-8522-8e32826a08c0\" (UID: \"34498079-d1c4-45ea-8522-8e32826a08c0\") "
Sep 07 16:21:52.195614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195500    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.196000 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:21:52.196000 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.195840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.196799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.196219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.196919 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.196821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.197250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.197202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.197635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.197412    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.197635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.197601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:21:52.198365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.198336    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.198689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.198665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-kube-api-access-2g4l7" (OuterVolumeSpecName: "kube-api-access-2g4l7") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "kube-api-access-2g4l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:21:52.198816 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.198792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.199256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.199222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-config" (OuterVolumeSpecName: "config") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.199421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.199375    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.199557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.199538    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.199707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.199684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:21:52.200132 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.200096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.200236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.200214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.200283 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.200250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.200665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.200651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-config-out" (OuterVolumeSpecName: "config-out") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:21:52.209504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.209485    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-web-config" (OuterVolumeSpecName: "web-config") pod "34498079-d1c4-45ea-8522-8e32826a08c0" (UID: "34498079-d1c4-45ea-8522-8e32826a08c0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:21:52.219118 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219097    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2" exitCode=0
Sep 07 16:21:52.219118 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219118    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc" exitCode=0
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219124    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b" exitCode=0
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219130    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19" exitCode=0
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219134    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146" exitCode=0
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219139    2577 generic.go:356] "Generic (PLEG): container finished" podID="34498079-d1c4-45ea-8522-8e32826a08c0" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd" exitCode=0
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"}
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"}
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"}
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"}
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219313    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"}
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"}
Sep 07 16:21:52.219404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.219371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34498079-d1c4-45ea-8522-8e32826a08c0","Type":"ContainerDied","Data":"e4aec86878530491053ae80b46152518785a7050527611cefae19c377f852de3"}
Sep 07 16:21:52.227842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.227823    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.234885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.234871    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.241940 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.241695    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.243287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.243264    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:21:52.246877 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.246859    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:21:52.248798 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.248782    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.255378 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.255357    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.262555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.262537    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.269216 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.269194    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.269581 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.269556    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.269818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.269593    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"} err="failed to get container status \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist"
Sep 07 16:21:52.269818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.269625    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.269987 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.269964    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.270060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.269997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"} err="failed to get container status \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist"
Sep 07 16:21:52.270060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.270019    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.270426 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.270402    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.270490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.270434    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"} err="failed to get container status \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist"
Sep 07 16:21:52.270490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.270458    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.270772 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.270735    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.270884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.270781    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"} err="failed to get container status \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist"
Sep 07 16:21:52.270884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.270796    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.271083 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.271068    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.271124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271086    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"} err="failed to get container status \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist"
Sep 07 16:21:52.271124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271099    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.271362 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.271342    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.271397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271370    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"} err="failed to get container status \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist"
Sep 07 16:21:52.271397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271391    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.271639 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271624    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:21:52.271716 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:21:52.271619    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.271747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271722    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"} err="failed to get container status \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist"
Sep 07 16:21:52.271747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271736    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.271993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271976    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-web"
Sep 07 16:21:52.272044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.271996    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-web"
Sep 07 16:21:52.272044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272007    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy"
Sep 07 16:21:52.272044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272016    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy"
Sep 07 16:21:52.272044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272028    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="config-reloader"
Sep 07 16:21:52.272044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272035    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="config-reloader"
Sep 07 16:21:52.272044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272045    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-thanos"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272051    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-thanos"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="prometheus"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272065    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="prometheus"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272070    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="thanos-sidecar"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272075    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="thanos-sidecar"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="init-config-reloader"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272086    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="init-config-reloader"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272041    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"} err="failed to get container status \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272121    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272132    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="thanos-sidecar"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272138    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-web"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272147    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="prometheus"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272152    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy-thanos"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272159    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="config-reloader"
Sep 07 16:21:52.272208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272166    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" containerName="kube-rbac-proxy"
Sep 07 16:21:52.272816 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272343    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"} err="failed to get container status \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist"
Sep 07 16:21:52.272816 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272356    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.272816 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272611    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"} err="failed to get container status \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist"
Sep 07 16:21:52.272816 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272632    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.273027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272944    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"} err="failed to get container status \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist"
Sep 07 16:21:52.273027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.272977    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.273315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.273293    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"} err="failed to get container status \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist"
Sep 07 16:21:52.273399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.273316    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.273613 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.273594    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"} err="failed to get container status \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist"
Sep 07 16:21:52.273668 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.273614    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.273928 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.273907    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"} err="failed to get container status \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist"
Sep 07 16:21:52.273928 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.273928    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.274220 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274199    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"} err="failed to get container status \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist"
Sep 07 16:21:52.274282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274224    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.274482 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274464    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"} err="failed to get container status \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist"
Sep 07 16:21:52.274520 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274483    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.274714 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274698    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"} err="failed to get container status \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist"
Sep 07 16:21:52.274777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274716    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.274978 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274963    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"} err="failed to get container status \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist"
Sep 07 16:21:52.274978 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.274979    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.275192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275175    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"} err="failed to get container status \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist"
Sep 07 16:21:52.275229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275193    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.275409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275390    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"} err="failed to get container status \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist"
Sep 07 16:21:52.275409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275406    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.275609 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275593    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"} err="failed to get container status \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist"
Sep 07 16:21:52.275673 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275610    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.275862 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275837    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"} err="failed to get container status \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist"
Sep 07 16:21:52.275905 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.275862    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.276040 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.276104 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276053    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"} err="failed to get container status \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist"
Sep 07 16:21:52.276104 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276073    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.276361 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276335    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"} err="failed to get container status \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist"
Sep 07 16:21:52.276479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276365    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.276747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276701    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"} err="failed to get container status \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist"
Sep 07 16:21:52.276818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.276766    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.277051 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277031    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"} err="failed to get container status \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist"
Sep 07 16:21:52.277119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277054    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.277314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277293    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"} err="failed to get container status \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist"
Sep 07 16:21:52.277359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277317    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.277598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277579    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"} err="failed to get container status \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist"
Sep 07 16:21:52.277675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277599    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.277857 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277840    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"} err="failed to get container status \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist"
Sep 07 16:21:52.277857 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.277857    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.278078 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278062    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"} err="failed to get container status \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist"
Sep 07 16:21:52.278134 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278079    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.278346 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278323    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"} err="failed to get container status \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist"
Sep 07 16:21:52.278421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278346    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.278421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 16:21:52.278523 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278507    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 16:21:52.279022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.278982    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"} err="failed to get container status \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist"
Sep 07 16:21:52.279111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.279029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 16:21:52.279111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.279034    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.279216 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.279107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 16:21:52.279990 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.279961    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"} err="failed to get container status \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist"
Sep 07 16:21:52.280082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.279991    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.281603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.280786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 16:21:52.281603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.281177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 16:21:52.281603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.281413    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 16:21:52.281819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.281728    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 16:21:52.282167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.282144    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 16:21:52.282167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.282160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r8lgc\""
Sep 07 16:21:52.282417 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.282400    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 16:21:52.282669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.282652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 16:21:52.282905 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.282887    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-69jbg0nev7r97\""
Sep 07 16:21:52.283854 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.283833    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 16:21:52.285028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.285001    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"} err="failed to get container status \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist"
Sep 07 16:21:52.285028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.285026    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.285381 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.285350    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"} err="failed to get container status \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist"
Sep 07 16:21:52.285464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.285399    2577 scope.go:117] "RemoveContainer" containerID="50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"
Sep 07 16:21:52.286364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.286332    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2"} err="failed to get container status \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": rpc error: code = NotFound desc = could not find container \"50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2\": container with ID starting with 50d679fe7637f52a00ba99de88a6714920d1f236e8bd50fb2dbfec90bd6fb9e2 not found: ID does not exist"
Sep 07 16:21:52.286364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.286359    2577 scope.go:117] "RemoveContainer" containerID="61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"
Sep 07 16:21:52.286930 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.286732    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc"} err="failed to get container status \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": rpc error: code = NotFound desc = could not find container \"61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc\": container with ID starting with 61b9d10a5cb87653afecaf0ed28e5860c73c709bf8664e8a01d443610cb850fc not found: ID does not exist"
Sep 07 16:21:52.286930 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.286774    2577 scope.go:117] "RemoveContainer" containerID="8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"
Sep 07 16:21:52.287999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.287827    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b"} err="failed to get container status \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": rpc error: code = NotFound desc = could not find container \"8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b\": container with ID starting with 8dce92102f19416fb63f991e8115fe780616b670b4318e102b26d6c72477ba3b not found: ID does not exist"
Sep 07 16:21:52.287999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.287861    2577 scope.go:117] "RemoveContainer" containerID="86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"
Sep 07 16:21:52.288494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.288171    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19"} err="failed to get container status \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": rpc error: code = NotFound desc = could not find container \"86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19\": container with ID starting with 86c27cb7a306c057e906660908d87848eb27ebbe9de9ef59cc873b5f59e4cc19 not found: ID does not exist"
Sep 07 16:21:52.288494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.288197    2577 scope.go:117] "RemoveContainer" containerID="495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"
Sep 07 16:21:52.288598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.288557    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146"} err="failed to get container status \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": rpc error: code = NotFound desc = could not find container \"495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146\": container with ID starting with 495afcaebe605a319e325db3d6ae69e3380c7b7a64e0ac983c7b8afc7826c146 not found: ID does not exist"
Sep 07 16:21:52.288598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.288581    2577 scope.go:117] "RemoveContainer" containerID="c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"
Sep 07 16:21:52.289252 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.289225    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 16:21:52.289574 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.289546    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd"} err="failed to get container status \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": rpc error: code = NotFound desc = could not find container \"c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd\": container with ID starting with c835146e27197ae75204986d704b40c8fdf47d9d9c193946adc222b4a98900fd not found: ID does not exist"
Sep 07 16:21:52.289629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.289577    2577 scope.go:117] "RemoveContainer" containerID="0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"
Sep 07 16:21:52.289885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.289864    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57"} err="failed to get container status \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": rpc error: code = NotFound desc = could not find container \"0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57\": container with ID starting with 0ea16aba67c4345dfa9c9f7c48981f255471507da066b42b2dfba802f7fd5d57 not found: ID does not exist"
Sep 07 16:21:52.289958 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.289914    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 16:21:52.290119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.290104    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:21:52.296229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296209    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-grpc-tls\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296230    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-metrics-client-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296240    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296249    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296258    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296266    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296274    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296282    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296290    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296298    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4l7\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-kube-api-access-2g4l7\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296306    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296314    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-db\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296329    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34498079-d1c4-45ea-8522-8e32826a08c0-config-out\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296337    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34498079-d1c4-45ea-8522-8e32826a08c0-tls-assets\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296344    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-kube-rbac-proxy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296352    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-secret-metrics-client-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296359    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34498079-d1c4-45ea-8522-8e32826a08c0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296367    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.296559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.296392    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34498079-d1c4-45ea-8522-8e32826a08c0-web-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:21:52.397559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.397559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.397811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c8709d2-fd14-4dfb-987f-b353237b61ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.397811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c8709d2-fd14-4dfb-987f-b353237b61ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.397811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.397811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.397811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.397993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398180 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-config\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398180 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398180 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398374 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398374 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.398436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.398424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whlmq\" (UniqueName: \"kubernetes.io/projected/8c8709d2-fd14-4dfb-987f-b353237b61ec-kube-api-access-whlmq\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whlmq\" (UniqueName: \"kubernetes.io/projected/8c8709d2-fd14-4dfb-987f-b353237b61ec-kube-api-access-whlmq\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c8709d2-fd14-4dfb-987f-b353237b61ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c8709d2-fd14-4dfb-987f-b353237b61ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-config\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.499544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.499705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.500218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.500590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.500563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.501083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.501057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.501691 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.501666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.502483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.502483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.502483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c8709d2-fd14-4dfb-987f-b353237b61ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.502483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c8709d2-fd14-4dfb-987f-b353237b61ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.503212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.503212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.503212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.502917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.503212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.503158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-config\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.504127 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.504085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.504447 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.504419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.504873 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.504856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.505191 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.505168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.505191 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.505181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c8709d2-fd14-4dfb-987f-b353237b61ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.505943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.505924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c8709d2-fd14-4dfb-987f-b353237b61ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.506722 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.506707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whlmq\" (UniqueName: \"kubernetes.io/projected/8c8709d2-fd14-4dfb-987f-b353237b61ec-kube-api-access-whlmq\") pod \"prometheus-k8s-0\" (UID: \"8c8709d2-fd14-4dfb-987f-b353237b61ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.593243 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.593209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:52.871253 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:52.871022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 16:21:52.873492 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:21:52.873468    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8709d2_fd14_4dfb_987f_b353237b61ec.slice/crio-898855525a6f415b8655e564b823437ad81a5e3b3c10ce79cdf5d89bed5d73a0 WatchSource:0}: Error finding container 898855525a6f415b8655e564b823437ad81a5e3b3c10ce79cdf5d89bed5d73a0: Status 404 returned error can't find the container with id 898855525a6f415b8655e564b823437ad81a5e3b3c10ce79cdf5d89bed5d73a0
Sep 07 16:21:53.225902 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:53.225865    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c8709d2-fd14-4dfb-987f-b353237b61ec" containerID="4b9ad0325fe124ce2b37791718207595a86e592ab9100a1e95c2cdaf3bf48403" exitCode=0
Sep 07 16:21:53.226257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:53.225948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerDied","Data":"4b9ad0325fe124ce2b37791718207595a86e592ab9100a1e95c2cdaf3bf48403"}
Sep 07 16:21:53.226257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:53.226010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"898855525a6f415b8655e564b823437ad81a5e3b3c10ce79cdf5d89bed5d73a0"}
Sep 07 16:21:53.572231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:53.572198    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34498079-d1c4-45ea-8522-8e32826a08c0" path="/var/lib/kubelet/pods/34498079-d1c4-45ea-8522-8e32826a08c0/volumes"
Sep 07 16:21:54.235497 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.235464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"5b1e56f0b947a233b300149fe94d466076b8afe228d806a7c85938ede203ebfe"}
Sep 07 16:21:54.235882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.235530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"8fce73e445a1932971ac508dda920c856303c91304dce97258f955b3e16e98ca"}
Sep 07 16:21:54.235882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.235554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"40eb9f896dfd341a938e59c87a9ae586d44c71a5fcae78f0881711e3482059d9"}
Sep 07 16:21:54.235882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.235577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"9d811c6a2db5e535a9fe4de02f1e2411b81b025655d65b1b788c5345c6d653fe"}
Sep 07 16:21:54.235882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.235599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"aab3f8afeaa978852a6b1b54f9b6518a4181110cc43055bd870138547f92d998"}
Sep 07 16:21:54.235882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.235639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c8709d2-fd14-4dfb-987f-b353237b61ec","Type":"ContainerStarted","Data":"70985eec8609a893b21ebee046207fe9952c1ec2038b75e4dd58c41b838026cf"}
Sep 07 16:21:54.263610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:54.263556    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.24678547 podStartE2EDuration="2.2635395s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.75403ms" isStatefulPod=false podCreationTimestamp="2026-09-07 16:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:21:54.262290774 +0000 UTC m=+119.143827743" watchObservedRunningTime="2026-09-07 16:21:54.2635395 +0000 UTC m=+119.145076493"
Sep 07 16:21:55.306811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:55.306773    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:55.359351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:55.359312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:21:57.593867 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:57.593835    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:21:59.935108 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:21:59.935062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:00.042335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:00.042292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:00.120357 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:00.120313    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:03.481101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:03.481070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:06.666510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:06.666474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:06.767621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:06.767579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:35.391004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:35.390964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:35.491714 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:35.491679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:52.593665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:52.593631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:22:52.610020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:52.609996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:22:53.483419 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:53.483391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 16:22:56.933866 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:56.933834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:22:57.710252 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:22:57.710219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:00.538596 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:00.538565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:00.588177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:00.588130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:02.962286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:02.962252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:03.065274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.065226    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:03.150085 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.150056    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mjqrk"]
Sep 07 16:23:03.153021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.153002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.155383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.155364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 16:23:03.156514 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.156493    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-95krs\""
Sep 07 16:23:03.156611 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.156513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 16:23:03.163549 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.163524    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mjqrk"]
Sep 07 16:23:03.193188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.193151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8db4fe74-479f-4040-86da-ec244e58f02a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mjqrk\" (UID: \"8db4fe74-479f-4040-86da-ec244e58f02a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.193372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.193201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/8db4fe74-479f-4040-86da-ec244e58f02a-kube-api-access-74x4g\") pod \"cert-manager-webhook-6486ffd997-mjqrk\" (UID: \"8db4fe74-479f-4040-86da-ec244e58f02a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.294585 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.294512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8db4fe74-479f-4040-86da-ec244e58f02a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mjqrk\" (UID: \"8db4fe74-479f-4040-86da-ec244e58f02a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.294585 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.294550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/8db4fe74-479f-4040-86da-ec244e58f02a-kube-api-access-74x4g\") pod \"cert-manager-webhook-6486ffd997-mjqrk\" (UID: \"8db4fe74-479f-4040-86da-ec244e58f02a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.306130 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.306106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/8db4fe74-479f-4040-86da-ec244e58f02a-kube-api-access-74x4g\") pod \"cert-manager-webhook-6486ffd997-mjqrk\" (UID: \"8db4fe74-479f-4040-86da-ec244e58f02a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.308957 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.308934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8db4fe74-479f-4040-86da-ec244e58f02a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mjqrk\" (UID: \"8db4fe74-479f-4040-86da-ec244e58f02a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.474925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.474893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:03.483557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.483524    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:03.603573 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.603483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:03.754274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:03.754250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mjqrk"]
Sep 07 16:23:03.756813 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:23:03.756782    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8db4fe74_479f_4040_86da_ec244e58f02a.slice/crio-a3dfb7fea9cee055499d79afb2cd4ddca20fc9cd6024c19630d4075f972a4c46 WatchSource:0}: Error finding container a3dfb7fea9cee055499d79afb2cd4ddca20fc9cd6024c19630d4075f972a4c46: Status 404 returned error can't find the container with id a3dfb7fea9cee055499d79afb2cd4ddca20fc9cd6024c19630d4075f972a4c46
Sep 07 16:23:04.513794 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:04.513738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk" event={"ID":"8db4fe74-479f-4040-86da-ec244e58f02a","Type":"ContainerStarted","Data":"a3dfb7fea9cee055499d79afb2cd4ddca20fc9cd6024c19630d4075f972a4c46"}
Sep 07 16:23:05.190145 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:05.190112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:05.363831 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:05.363789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:07.340325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:07.340266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:07.394005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:07.393977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:07.529576 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:07.529487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk" event={"ID":"8db4fe74-479f-4040-86da-ec244e58f02a","Type":"ContainerStarted","Data":"b086b535052c9c6f948eae09978525c7b4c180abbdc9b69af930fdd93f9885c1"}
Sep 07 16:23:07.529740 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:07.529647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:07.544630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:07.544583    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk" podStartSLOduration=1.1573274709999999 podStartE2EDuration="4.544568527s" totalImagesPullingTime="3.387241056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:07.543304451 +0000 UTC m=+192.424841418" watchObservedRunningTime="2026-09-07 16:23:07.544568527 +0000 UTC m=+192.426105496"
Sep 07 16:23:10.284833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:10.284791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:10.342038 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:10.341973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:12.971605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:12.971572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:13.535778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:13.535733    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mjqrk"
Sep 07 16:23:14.624864 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.624829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tv4hp"]
Sep 07 16:23:14.627138 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.627119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.629710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.629689    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6ljv8\""
Sep 07 16:23:14.638781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.638742    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tv4hp"]
Sep 07 16:23:14.705547 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.705510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4nh\" (UniqueName: \"kubernetes.io/projected/01390eba-e40b-430c-a7e4-34c143f8a87f-kube-api-access-9w4nh\") pod \"cert-manager-55b4d954d5-tv4hp\" (UID: \"01390eba-e40b-430c-a7e4-34c143f8a87f\") " pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.705716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.705568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01390eba-e40b-430c-a7e4-34c143f8a87f-bound-sa-token\") pod \"cert-manager-55b4d954d5-tv4hp\" (UID: \"01390eba-e40b-430c-a7e4-34c143f8a87f\") " pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.806918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.806862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4nh\" (UniqueName: \"kubernetes.io/projected/01390eba-e40b-430c-a7e4-34c143f8a87f-kube-api-access-9w4nh\") pod \"cert-manager-55b4d954d5-tv4hp\" (UID: \"01390eba-e40b-430c-a7e4-34c143f8a87f\") " pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.807092 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.806945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01390eba-e40b-430c-a7e4-34c143f8a87f-bound-sa-token\") pod \"cert-manager-55b4d954d5-tv4hp\" (UID: \"01390eba-e40b-430c-a7e4-34c143f8a87f\") " pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.815271 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.815230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01390eba-e40b-430c-a7e4-34c143f8a87f-bound-sa-token\") pod \"cert-manager-55b4d954d5-tv4hp\" (UID: \"01390eba-e40b-430c-a7e4-34c143f8a87f\") " pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.815391 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.815330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4nh\" (UniqueName: \"kubernetes.io/projected/01390eba-e40b-430c-a7e4-34c143f8a87f-kube-api-access-9w4nh\") pod \"cert-manager-55b4d954d5-tv4hp\" (UID: \"01390eba-e40b-430c-a7e4-34c143f8a87f\") " pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:14.935914 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:14.935880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tv4hp"
Sep 07 16:23:15.201448 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:15.201258    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tv4hp"]
Sep 07 16:23:15.204166 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:23:15.204141    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01390eba_e40b_430c_a7e4_34c143f8a87f.slice/crio-177a01c37b3f28705ff5951532d6e21113f3ba16dbf954374d9deb6d3a2b37fa WatchSource:0}: Error finding container 177a01c37b3f28705ff5951532d6e21113f3ba16dbf954374d9deb6d3a2b37fa: Status 404 returned error can't find the container with id 177a01c37b3f28705ff5951532d6e21113f3ba16dbf954374d9deb6d3a2b37fa
Sep 07 16:23:15.567341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:15.567260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tv4hp" event={"ID":"01390eba-e40b-430c-a7e4-34c143f8a87f","Type":"ContainerStarted","Data":"877939c3b5cb296d7f884e7cb64ec084317ad40369978dd6b9239a4090c8cf39"}
Sep 07 16:23:15.567341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:15.567296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tv4hp" event={"ID":"01390eba-e40b-430c-a7e4-34c143f8a87f","Type":"ContainerStarted","Data":"177a01c37b3f28705ff5951532d6e21113f3ba16dbf954374d9deb6d3a2b37fa"}
Sep 07 16:23:15.582611 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:15.582557    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tv4hp" podStartSLOduration=1.582541429 podStartE2EDuration="1.582541429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:15.580992209 +0000 UTC m=+200.462529179" watchObservedRunningTime="2026-09-07 16:23:15.582541429 +0000 UTC m=+200.464078396"
Sep 07 16:23:16.320036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:16.320006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:18.753062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:18.753027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:18.856554 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:18.856521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:24.975133 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:24.975092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:25.072673 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:25.072637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:27.256341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:27.256311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:29.506250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:29.506210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:29.552938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:29.552908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:39.457205 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.457173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:39.510838 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.510807    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"]
Sep 07 16:23:39.517067 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.517040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.520822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.520800    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 16:23:39.521876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.521848    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 16:23:39.522004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.521856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 16:23:39.522224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.522209    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q76fw\""
Sep 07 16:23:39.522285 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.522214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 16:23:39.522285 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.522241    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 16:23:39.529486 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.529466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"]
Sep 07 16:23:39.531025 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.531002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afc46a41-520b-4a4b-abf3-33d54eb9dc25-cert\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.531149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.531072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/afc46a41-520b-4a4b-abf3-33d54eb9dc25-metrics-cert\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.531246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.531175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afc46a41-520b-4a4b-abf3-33d54eb9dc25-manager-config\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.531246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.531216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ccq\" (UniqueName: \"kubernetes.io/projected/afc46a41-520b-4a4b-abf3-33d54eb9dc25-kube-api-access-26ccq\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.560162 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.560120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:39.632565 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.632522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afc46a41-520b-4a4b-abf3-33d54eb9dc25-manager-config\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.632565 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.632567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ccq\" (UniqueName: \"kubernetes.io/projected/afc46a41-520b-4a4b-abf3-33d54eb9dc25-kube-api-access-26ccq\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.632833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.632690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afc46a41-520b-4a4b-abf3-33d54eb9dc25-cert\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.632833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.632787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/afc46a41-520b-4a4b-abf3-33d54eb9dc25-metrics-cert\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.633294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.633273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afc46a41-520b-4a4b-abf3-33d54eb9dc25-manager-config\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.635313 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.635290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/afc46a41-520b-4a4b-abf3-33d54eb9dc25-metrics-cert\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.635458 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.635315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afc46a41-520b-4a4b-abf3-33d54eb9dc25-cert\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.639731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.639709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ccq\" (UniqueName: \"kubernetes.io/projected/afc46a41-520b-4a4b-abf3-33d54eb9dc25-kube-api-access-26ccq\") pod \"lws-controller-manager-57d49c86b4-2pmxs\" (UID: \"afc46a41-520b-4a4b-abf3-33d54eb9dc25\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:39.826795 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:39.826687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:40.136880 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:40.136848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"]
Sep 07 16:23:40.140903 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:23:40.140877    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc46a41_520b_4a4b_abf3_33d54eb9dc25.slice/crio-29ff221f34a6a4b8ef9b57297adbde56c344c6c4a2a4478ea495050e8c87a49a WatchSource:0}: Error finding container 29ff221f34a6a4b8ef9b57297adbde56c344c6c4a2a4478ea495050e8c87a49a: Status 404 returned error can't find the container with id 29ff221f34a6a4b8ef9b57297adbde56c344c6c4a2a4478ea495050e8c87a49a
Sep 07 16:23:40.671747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:40.671715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs" event={"ID":"afc46a41-520b-4a4b-abf3-33d54eb9dc25","Type":"ContainerStarted","Data":"29ff221f34a6a4b8ef9b57297adbde56c344c6c4a2a4478ea495050e8c87a49a"}
Sep 07 16:23:43.362413 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:43.362370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:43.484117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:43.484088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:43.690280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:43.690249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs" event={"ID":"afc46a41-520b-4a4b-abf3-33d54eb9dc25","Type":"ContainerStarted","Data":"a5ffa1f1712b6c9cfca6d8e25857661e976d2efb4746a6f0d73e468d2737bc52"}
Sep 07 16:23:43.690469 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:43.690387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:43.707611 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:43.707437    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs" podStartSLOduration=2.126969361 podStartE2EDuration="4.707417783s" totalImagesPullingTime="2.580448422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:43.707014868 +0000 UTC m=+228.588551840" watchObservedRunningTime="2026-09-07 16:23:43.707417783 +0000 UTC m=+228.588954750"
Sep 07 16:23:44.161404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.161363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:44.390744 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.390710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:44.493562 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.493476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:44.517541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.517513    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"]
Sep 07 16:23:44.520978 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.520962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.524556 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.524529    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 16:23:44.524699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.524529    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f6lx2\""
Sep 07 16:23:44.524699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.524637    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 16:23:44.524699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.524529    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 16:23:44.524906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.524888    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 16:23:44.538002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.537977    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"]
Sep 07 16:23:44.577834 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.577800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fab5e120-7b65-420c-a1ab-84cc4f202814-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.577997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.577854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fab5e120-7b65-420c-a1ab-84cc4f202814-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.577997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.577923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab5e120-7b65-420c-a1ab-84cc4f202814-tmp\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.577997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.577982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdcw7\" (UniqueName: \"kubernetes.io/projected/fab5e120-7b65-420c-a1ab-84cc4f202814-kube-api-access-kdcw7\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.578091 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.578027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fab5e120-7b65-420c-a1ab-84cc4f202814-manifests\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.626350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.626304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:44.678941 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.678899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fab5e120-7b65-420c-a1ab-84cc4f202814-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.679110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.678960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab5e120-7b65-420c-a1ab-84cc4f202814-tmp\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.679110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.679029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdcw7\" (UniqueName: \"kubernetes.io/projected/fab5e120-7b65-420c-a1ab-84cc4f202814-kube-api-access-kdcw7\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.679110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.679065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fab5e120-7b65-420c-a1ab-84cc4f202814-manifests\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.679110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.679107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fab5e120-7b65-420c-a1ab-84cc4f202814-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.679613 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.679584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fab5e120-7b65-420c-a1ab-84cc4f202814-manifests\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.681898 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.681869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fab5e120-7b65-420c-a1ab-84cc4f202814-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.681898 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.681895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab5e120-7b65-420c-a1ab-84cc4f202814-tmp\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.682217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.682198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fab5e120-7b65-420c-a1ab-84cc4f202814-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.687100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.687074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdcw7\" (UniqueName: \"kubernetes.io/projected/fab5e120-7b65-420c-a1ab-84cc4f202814-kube-api-access-kdcw7\") pod \"opendatahub-operator-controller-manager-7766cc5489-b67gx\" (UID: \"fab5e120-7b65-420c-a1ab-84cc4f202814\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.733127 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.733093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:44.826255 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.826211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:44.831997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.831971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:44.928160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:44.928128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:45.038335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:45.038304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:45.114967 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:45.114942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"]
Sep 07 16:23:45.117446 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:23:45.117418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab5e120_7b65_420c_a1ab_84cc4f202814.slice/crio-b9fbe52c050babf085915bf113dd607b7bed826f0181a7ee86e2e84b0096b5db WatchSource:0}: Error finding container b9fbe52c050babf085915bf113dd607b7bed826f0181a7ee86e2e84b0096b5db: Status 404 returned error can't find the container with id b9fbe52c050babf085915bf113dd607b7bed826f0181a7ee86e2e84b0096b5db
Sep 07 16:23:45.119004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:45.118972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:45.700728 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:45.700694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx" event={"ID":"fab5e120-7b65-420c-a1ab-84cc4f202814","Type":"ContainerStarted","Data":"b9fbe52c050babf085915bf113dd607b7bed826f0181a7ee86e2e84b0096b5db"}
Sep 07 16:23:48.719199 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:48.719161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx" event={"ID":"fab5e120-7b65-420c-a1ab-84cc4f202814","Type":"ContainerStarted","Data":"59d2d5021a3b80397f4a6f0c8421d3ee1795f821f94dc143defd10a02c769eed"}
Sep 07 16:23:49.725037 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:49.724958    2577 generic.go:356] "Generic (PLEG): container finished" podID="fab5e120-7b65-420c-a1ab-84cc4f202814" containerID="59d2d5021a3b80397f4a6f0c8421d3ee1795f821f94dc143defd10a02c769eed" exitCode=0
Sep 07 16:23:49.725446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:49.725038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx" event={"ID":"fab5e120-7b65-420c-a1ab-84cc4f202814","Type":"ContainerDied","Data":"59d2d5021a3b80397f4a6f0c8421d3ee1795f821f94dc143defd10a02c769eed"}
Sep 07 16:23:50.731876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:50.731839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx" event={"ID":"fab5e120-7b65-420c-a1ab-84cc4f202814","Type":"ContainerStarted","Data":"cae4f7506bc15b698e6739dcadacf4e217911cc10e687b8cc5435d01e2a155ef"}
Sep 07 16:23:50.732340 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:50.732019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:23:50.753903 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:50.753854    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx" podStartSLOduration=2.217896746 podStartE2EDuration="6.753838564s" totalImagesPullingTime="3.525540841s" totalInitContainerRuntime="1.010400977s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:50.751412627 +0000 UTC m=+235.632949605" watchObservedRunningTime="2026-09-07 16:23:50.753838564 +0000 UTC m=+235.635375532"
Sep 07 16:23:54.698472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:54.698444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-2pmxs"
Sep 07 16:23:56.363498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:56.363465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:23:56.463575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:23:56.463539    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:01.738887 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:01.738862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-b67gx"
Sep 07 16:24:03.822403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:03.822369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:03.982856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:03.982820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:04.192429 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.192397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:04.246515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.246477    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"]
Sep 07 16:24:04.255973 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.255946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.258664 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.258641    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 16:24:04.259051 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.259030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"]
Sep 07 16:24:04.259787 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.259745    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8xchv\""
Sep 07 16:24:04.259885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.259746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 16:24:04.259885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.259854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 16:24:04.259885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.259880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 16:24:04.353356 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.353318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0478b80e-6252-47e7-8559-4bff15e9d972-tmp\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.353551 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.353412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0478b80e-6252-47e7-8559-4bff15e9d972-tls-certs\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.353551 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.353468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68khw\" (UniqueName: \"kubernetes.io/projected/0478b80e-6252-47e7-8559-4bff15e9d972-kube-api-access-68khw\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.454701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.454613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0478b80e-6252-47e7-8559-4bff15e9d972-tls-certs\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.454701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.454674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68khw\" (UniqueName: \"kubernetes.io/projected/0478b80e-6252-47e7-8559-4bff15e9d972-kube-api-access-68khw\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.454954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.454733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0478b80e-6252-47e7-8559-4bff15e9d972-tmp\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.454954 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:24:04.454811    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 07 16:24:04.454954 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:24:04.454886    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0478b80e-6252-47e7-8559-4bff15e9d972-tls-certs podName:0478b80e-6252-47e7-8559-4bff15e9d972 nodeName:}" failed. No retries permitted until 2026-09-07 16:24:04.95486783 +0000 UTC m=+249.836404978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0478b80e-6252-47e7-8559-4bff15e9d972-tls-certs") pod "kube-auth-proxy-6797cb654d-6kdxj" (UID: "0478b80e-6252-47e7-8559-4bff15e9d972") : secret "kube-auth-proxy-tls" not found
Sep 07 16:24:04.457077 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.457059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0478b80e-6252-47e7-8559-4bff15e9d972-tmp\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.465716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.465692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68khw\" (UniqueName: \"kubernetes.io/projected/0478b80e-6252-47e7-8559-4bff15e9d972-kube-api-access-68khw\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.859835 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.859733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:04.964268 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.964224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0478b80e-6252-47e7-8559-4bff15e9d972-tls-certs\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:04.966748 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:04.966724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0478b80e-6252-47e7-8559-4bff15e9d972-tls-certs\") pod \"kube-auth-proxy-6797cb654d-6kdxj\" (UID: \"0478b80e-6252-47e7-8559-4bff15e9d972\") " pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:05.166365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:05.166326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"
Sep 07 16:24:05.431059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:05.430981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:05.438027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:05.438002    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj"]
Sep 07 16:24:05.442041 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:24:05.442017    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0478b80e_6252_47e7_8559_4bff15e9d972.slice/crio-0ba524594ee7e5bc94105079eda34315bbe659704a398f78fb237464a6d72a2b WatchSource:0}: Error finding container 0ba524594ee7e5bc94105079eda34315bbe659704a398f78fb237464a6d72a2b: Status 404 returned error can't find the container with id 0ba524594ee7e5bc94105079eda34315bbe659704a398f78fb237464a6d72a2b
Sep 07 16:24:05.491343 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:05.491311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:05.799930 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:05.799839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj" event={"ID":"0478b80e-6252-47e7-8559-4bff15e9d972","Type":"ContainerStarted","Data":"0ba524594ee7e5bc94105079eda34315bbe659704a398f78fb237464a6d72a2b"}
Sep 07 16:24:06.405645 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:06.405584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:08.816965 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:08.816924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj" event={"ID":"0478b80e-6252-47e7-8559-4bff15e9d972","Type":"ContainerStarted","Data":"082fd84bb14bc8ff25d5ace066661c5f6132c5de1a2e61cf7db2692a8f592135"}
Sep 07 16:24:08.833098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:08.833048    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6797cb654d-6kdxj" podStartSLOduration=2.198592342 podStartE2EDuration="4.833028676s" totalImagesPullingTime="2.634436334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:24:08.831072055 +0000 UTC m=+253.712609023" watchObservedRunningTime="2026-09-07 16:24:08.833028676 +0000 UTC m=+253.714565645"
Sep 07 16:24:10.461307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:10.461266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:10.526836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:10.526803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:11.036977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:11.036939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:11.239044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:11.239003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:11.404918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:11.404884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:14.878361 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:14.878327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:14.979147 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:14.979107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:15.079563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:15.079530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:15.180128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:15.180098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:15.409441 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:15.409399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:15.843474 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:15.843414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:23.955160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:23.955044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:25.329405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:25.329376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:25.391400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:25.391373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:24:55.488332 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:24:55.488287    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 16:25:15.886617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:15.886584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:15.981679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:15.981646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:28.536128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:28.536097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:31.324840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:31.324807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:33.121737 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:33.121705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:33.403646 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:33.403604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:33.553716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:33.553675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:34.132987 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:34.132936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:34.786435 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:34.786394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:35.788041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:35.788009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:35.889892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:35.889857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:37.150546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:37.150515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:37.368560 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:37.368528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:37.530201 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:37.530118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:37.636911 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:37.636880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:40.940305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:40.940262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:41.039504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:41.039470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:42.953372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:42.953335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:43.053605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:43.053574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:44.029072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:44.029041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:44.125626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:44.125594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:44.226767 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:44.226726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:48.516332 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:48.516244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:48.615361 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:48.615317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:48.718652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:48.718598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:54.096666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.096634    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"]
Sep 07 16:25:54.100074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.100057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.102600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.102571    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 16:25:54.102715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.102573    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 16:25:54.102715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.102573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z2dkf\""
Sep 07 16:25:54.113136 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.113108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"]
Sep 07 16:25:54.139669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.139624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.139857 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.139686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmg2m\" (UniqueName: \"kubernetes.io/projected/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-kube-api-access-tmg2m\") pod \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.230216 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.230181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"]
Sep 07 16:25:54.230478 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:25:54.230459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-tmg2m], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n" podUID="3a4e5d99-a7f1-4377-a06c-ad1699f214dd"
Sep 07 16:25:54.239892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.239861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:54.240123 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.240106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.240165 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.240136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmg2m\" (UniqueName: \"kubernetes.io/projected/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-kube-api-access-tmg2m\") pod \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.240491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.240475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.257874 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.257847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmg2m\" (UniqueName: \"kubernetes.io/projected/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-kube-api-access-tmg2m\") pod \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.317508 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.317473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.325342 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.325318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:54.330703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.330667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:54.341198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.341169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-extensions-socket-volume\") pod \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") "
Sep 07 16:25:54.341342 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.341254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmg2m\" (UniqueName: \"kubernetes.io/projected/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-kube-api-access-tmg2m\") pod \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\" (UID: \"3a4e5d99-a7f1-4377-a06c-ad1699f214dd\") "
Sep 07 16:25:54.341516 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.341490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3a4e5d99-a7f1-4377-a06c-ad1699f214dd" (UID: "3a4e5d99-a7f1-4377-a06c-ad1699f214dd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:25:54.343562 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.343534    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-kube-api-access-tmg2m" (OuterVolumeSpecName: "kube-api-access-tmg2m") pod "3a4e5d99-a7f1-4377-a06c-ad1699f214dd" (UID: "3a4e5d99-a7f1-4377-a06c-ad1699f214dd"). InnerVolumeSpecName "kube-api-access-tmg2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:25:54.429339 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.429312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:54.441876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.441852    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-extensions-socket-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:25:54.441876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.441875    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmg2m\" (UniqueName: \"kubernetes.io/projected/3a4e5d99-a7f1-4377-a06c-ad1699f214dd-kube-api-access-tmg2m\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:25:54.527037 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.527010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:54.994638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:54.994604    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"]
Sep 07 16:25:55.008331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.008301    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"]
Sep 07 16:25:55.040602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.040570    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"]
Sep 07 16:25:55.044091 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.044073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:25:55.046505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.046479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rx5s\" (UniqueName: \"kubernetes.io/projected/fc41bd89-0b91-440e-be33-151ae38f506b-kube-api-access-9rx5s\") pod \"limitador-operator-controller-manager-6c549b6d85-l6ssx\" (UID: \"fc41bd89-0b91-440e-be33-151ae38f506b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:25:55.046768 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.046737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-29jlw\""
Sep 07 16:25:55.057954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.057930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"]
Sep 07 16:25:55.146988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.146954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rx5s\" (UniqueName: \"kubernetes.io/projected/fc41bd89-0b91-440e-be33-151ae38f506b-kube-api-access-9rx5s\") pod \"limitador-operator-controller-manager-6c549b6d85-l6ssx\" (UID: \"fc41bd89-0b91-440e-be33-151ae38f506b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:25:55.158587 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.158555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rx5s\" (UniqueName: \"kubernetes.io/projected/fc41bd89-0b91-440e-be33-151ae38f506b-kube-api-access-9rx5s\") pod \"limitador-operator-controller-manager-6c549b6d85-l6ssx\" (UID: \"fc41bd89-0b91-440e-be33-151ae38f506b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:25:55.321665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.321583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:55.324804 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:25:55.324770    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="3a4e5d99-a7f1-4377-a06c-ad1699f214dd" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:55.330425 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:25:55.330395    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="3a4e5d99-a7f1-4377-a06c-ad1699f214dd" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:55.353655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.353614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:25:55.566993 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:25:55.566958    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-t7l6n\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="3a4e5d99-a7f1-4377-a06c-ad1699f214dd" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-t7l6n"
Sep 07 16:25:55.568810 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.568786    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a4e5d99-a7f1-4377-a06c-ad1699f214dd" path="/var/lib/kubelet/pods/3a4e5d99-a7f1-4377-a06c-ad1699f214dd/volumes"
Sep 07 16:25:55.638481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.638453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"]
Sep 07 16:25:55.641498 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:25:55.641476    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc41bd89_0b91_440e_be33_151ae38f506b.slice/crio-e17daa9113cff39885a8f5579918a153cf3b7fe1802cb2a6e9db1f131a373d3b WatchSource:0}: Error finding container e17daa9113cff39885a8f5579918a153cf3b7fe1802cb2a6e9db1f131a373d3b: Status 404 returned error can't find the container with id e17daa9113cff39885a8f5579918a153cf3b7fe1802cb2a6e9db1f131a373d3b
Sep 07 16:25:55.643548 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:55.643532    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 16:25:56.327721 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:56.327683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx" event={"ID":"fc41bd89-0b91-440e-be33-151ae38f506b","Type":"ContainerStarted","Data":"e17daa9113cff39885a8f5579918a153cf3b7fe1802cb2a6e9db1f131a373d3b"}
Sep 07 16:25:57.325415 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:57.325383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:57.420486 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:57.420458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:25:59.918136 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:25:59.918100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:00.018560 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:00.018521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:00.347160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:00.347079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx" event={"ID":"fc41bd89-0b91-440e-be33-151ae38f506b","Type":"ContainerStarted","Data":"a4038c9f2792241a6550062327c82aacc2d1088407c54d1da6feddd9dfdbf728"}
Sep 07 16:26:00.347305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:00.347161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:26:00.365119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:00.365067    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx" podStartSLOduration=1.720779541 podStartE2EDuration="5.365049996s" totalImagesPullingTime="3.644270455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:26:00.36350622 +0000 UTC m=+365.245043202" watchObservedRunningTime="2026-09-07 16:26:00.365049996 +0000 UTC m=+365.246586965"
Sep 07 16:26:03.111838 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:03.111804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:03.212365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:03.212329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:07.509090 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:07.509020    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:07.609767 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:07.609717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:07.712855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:07.712818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:07.773172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:07.773091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:08.802674 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:08.802636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:08.892934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:08.892907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:11.353912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:11.353881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-l6ssx"
Sep 07 16:26:16.321938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:16.321907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:16.373921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:16.373889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:24.389008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:24.388973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:24.483828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:24.483778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:37.111439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.111397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:37.219806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.219775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:37.273558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.273527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:37.891128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.891092    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-h76j7"]
Sep 07 16:26:37.894657 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.894638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:37.897128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.897105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhh4k\""
Sep 07 16:26:37.898213 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.898195    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-h76j7"]
Sep 07 16:26:37.924563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.924530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:37.995810 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:37.995780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:38.006946 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:38.006915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgj4z\" (UniqueName: \"kubernetes.io/projected/442eaa55-e3e9-4501-83f5-e381f1e1267e-kube-api-access-rgj4z\") pod \"authorino-b54df94b4-h76j7\" (UID: \"442eaa55-e3e9-4501-83f5-e381f1e1267e\") " pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:38.108122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:38.108081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgj4z\" (UniqueName: \"kubernetes.io/projected/442eaa55-e3e9-4501-83f5-e381f1e1267e-kube-api-access-rgj4z\") pod \"authorino-b54df94b4-h76j7\" (UID: \"442eaa55-e3e9-4501-83f5-e381f1e1267e\") " pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:38.115982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:38.115951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgj4z\" (UniqueName: \"kubernetes.io/projected/442eaa55-e3e9-4501-83f5-e381f1e1267e-kube-api-access-rgj4z\") pod \"authorino-b54df94b4-h76j7\" (UID: \"442eaa55-e3e9-4501-83f5-e381f1e1267e\") " pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:38.205170 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:38.205087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:38.471339 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:38.471316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-h76j7"]
Sep 07 16:26:38.473435 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:26:38.473410    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod442eaa55_e3e9_4501_83f5_e381f1e1267e.slice/crio-687281154d3c40cb1fbef7aa34ce1bc5c78b41c43596865fae1605d146183351 WatchSource:0}: Error finding container 687281154d3c40cb1fbef7aa34ce1bc5c78b41c43596865fae1605d146183351: Status 404 returned error can't find the container with id 687281154d3c40cb1fbef7aa34ce1bc5c78b41c43596865fae1605d146183351
Sep 07 16:26:38.525863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:38.525829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-h76j7" event={"ID":"442eaa55-e3e9-4501-83f5-e381f1e1267e","Type":"ContainerStarted","Data":"687281154d3c40cb1fbef7aa34ce1bc5c78b41c43596865fae1605d146183351"}
Sep 07 16:26:41.389654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:41.389619    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-h76j7"]
Sep 07 16:26:41.542667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:41.542627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-h76j7" event={"ID":"442eaa55-e3e9-4501-83f5-e381f1e1267e","Type":"ContainerStarted","Data":"45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2"}
Sep 07 16:26:41.557143 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:41.557096    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-h76j7" podStartSLOduration=2.499482452 podStartE2EDuration="4.55707994s" totalImagesPullingTime="2.057597488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:26:41.555233296 +0000 UTC m=+406.436770266" watchObservedRunningTime="2026-09-07 16:26:41.55707994 +0000 UTC m=+406.438616906"
Sep 07 16:26:42.547645 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:42.547608    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-h76j7" podUID="442eaa55-e3e9-4501-83f5-e381f1e1267e" containerName="authorino" containerID="cri-o://45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2" gracePeriod=30
Sep 07 16:26:42.789165 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:42.789142    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:42.953917 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:42.953890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgj4z\" (UniqueName: \"kubernetes.io/projected/442eaa55-e3e9-4501-83f5-e381f1e1267e-kube-api-access-rgj4z\") pod \"442eaa55-e3e9-4501-83f5-e381f1e1267e\" (UID: \"442eaa55-e3e9-4501-83f5-e381f1e1267e\") "
Sep 07 16:26:42.956120 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:42.956099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442eaa55-e3e9-4501-83f5-e381f1e1267e-kube-api-access-rgj4z" (OuterVolumeSpecName: "kube-api-access-rgj4z") pod "442eaa55-e3e9-4501-83f5-e381f1e1267e" (UID: "442eaa55-e3e9-4501-83f5-e381f1e1267e"). InnerVolumeSpecName "kube-api-access-rgj4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:26:43.055140 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.055101    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgj4z\" (UniqueName: \"kubernetes.io/projected/442eaa55-e3e9-4501-83f5-e381f1e1267e-kube-api-access-rgj4z\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:26:43.553292 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.553257    2577 generic.go:356] "Generic (PLEG): container finished" podID="442eaa55-e3e9-4501-83f5-e381f1e1267e" containerID="45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2" exitCode=0
Sep 07 16:26:43.553694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.553301    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-h76j7"
Sep 07 16:26:43.553694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.553329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-h76j7" event={"ID":"442eaa55-e3e9-4501-83f5-e381f1e1267e","Type":"ContainerDied","Data":"45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2"}
Sep 07 16:26:43.553694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.553366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-h76j7" event={"ID":"442eaa55-e3e9-4501-83f5-e381f1e1267e","Type":"ContainerDied","Data":"687281154d3c40cb1fbef7aa34ce1bc5c78b41c43596865fae1605d146183351"}
Sep 07 16:26:43.553694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.553383    2577 scope.go:117] "RemoveContainer" containerID="45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2"
Sep 07 16:26:43.562343 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.562319    2577 scope.go:117] "RemoveContainer" containerID="45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2"
Sep 07 16:26:43.562696 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:26:43.562672    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2\": container with ID starting with 45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2 not found: ID does not exist" containerID="45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2"
Sep 07 16:26:43.562792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.562709    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2"} err="failed to get container status \"45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2\": rpc error: code = NotFound desc = could not find container \"45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2\": container with ID starting with 45b38c287673e230ea9b948b93802d1c0a310c0e6fb26bb0eece0090450fa2f2 not found: ID does not exist"
Sep 07 16:26:43.573885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.573855    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-h76j7"]
Sep 07 16:26:43.577806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:43.577776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-h76j7"]
Sep 07 16:26:45.566822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:45.566789    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442eaa55-e3e9-4501-83f5-e381f1e1267e" path="/var/lib/kubelet/pods/442eaa55-e3e9-4501-83f5-e381f1e1267e/volumes"
Sep 07 16:26:55.513892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.513855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:55.610160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.610120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:55.955791 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.955744    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"]
Sep 07 16:26:55.956189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.956168    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442eaa55-e3e9-4501-83f5-e381f1e1267e" containerName="authorino"
Sep 07 16:26:55.956189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.956188    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="442eaa55-e3e9-4501-83f5-e381f1e1267e" containerName="authorino"
Sep 07 16:26:55.956404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.956259    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="442eaa55-e3e9-4501-83f5-e381f1e1267e" containerName="authorino"
Sep 07 16:26:55.960712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.960689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:55.967849 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.967773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 16:26:55.968328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.967986    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 07 16:26:55.968918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.968899    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-854bp\""
Sep 07 16:26:55.969019 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.968940    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 16:26:55.979812 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:55.979789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"]
Sep 07 16:26:56.039869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.039832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:26:56.064281 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.064254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/688bddf6-e99a-43b8-8b64-65cea3eae249-data\") pod \"postgres-69fb4594cf-k4rps\" (UID: \"688bddf6-e99a-43b8-8b64-65cea3eae249\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.064395 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.064338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6dcn\" (UniqueName: \"kubernetes.io/projected/688bddf6-e99a-43b8-8b64-65cea3eae249-kube-api-access-h6dcn\") pod \"postgres-69fb4594cf-k4rps\" (UID: \"688bddf6-e99a-43b8-8b64-65cea3eae249\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.165588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.165551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6dcn\" (UniqueName: \"kubernetes.io/projected/688bddf6-e99a-43b8-8b64-65cea3eae249-kube-api-access-h6dcn\") pod \"postgres-69fb4594cf-k4rps\" (UID: \"688bddf6-e99a-43b8-8b64-65cea3eae249\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.165783 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.165610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/688bddf6-e99a-43b8-8b64-65cea3eae249-data\") pod \"postgres-69fb4594cf-k4rps\" (UID: \"688bddf6-e99a-43b8-8b64-65cea3eae249\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.165989 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.165974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/688bddf6-e99a-43b8-8b64-65cea3eae249-data\") pod \"postgres-69fb4594cf-k4rps\" (UID: \"688bddf6-e99a-43b8-8b64-65cea3eae249\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.174325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.174307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6dcn\" (UniqueName: \"kubernetes.io/projected/688bddf6-e99a-43b8-8b64-65cea3eae249-kube-api-access-h6dcn\") pod \"postgres-69fb4594cf-k4rps\" (UID: \"688bddf6-e99a-43b8-8b64-65cea3eae249\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.272712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.272630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:26:56.541892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.541867    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"]
Sep 07 16:26:56.545838 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:26:56.545797    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688bddf6_e99a_43b8_8b64_65cea3eae249.slice/crio-3fd0712005ca71e5cb0fbc43dd36501d4299c562129b628af3dcec47e468abbb WatchSource:0}: Error finding container 3fd0712005ca71e5cb0fbc43dd36501d4299c562129b628af3dcec47e468abbb: Status 404 returned error can't find the container with id 3fd0712005ca71e5cb0fbc43dd36501d4299c562129b628af3dcec47e468abbb
Sep 07 16:26:56.613122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:56.613083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps" event={"ID":"688bddf6-e99a-43b8-8b64-65cea3eae249","Type":"ContainerStarted","Data":"3fd0712005ca71e5cb0fbc43dd36501d4299c562129b628af3dcec47e468abbb"}
Sep 07 16:26:59.922612 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:26:59.922572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:01.643238 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:01.643201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps" event={"ID":"688bddf6-e99a-43b8-8b64-65cea3eae249","Type":"ContainerStarted","Data":"395a3da18937cb51608cea01ae243bd1c3e1db44fa8e791dbce2ee2b609f4672"}
Sep 07 16:27:01.643623 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:01.643357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:27:01.658610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:01.658560    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps" podStartSLOduration=2.01600916 podStartE2EDuration="6.658544837s" totalImagesPullingTime="4.642535677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:27:01.656947667 +0000 UTC m=+426.538484636" watchObservedRunningTime="2026-09-07 16:27:01.658544837 +0000 UTC m=+426.540081805"
Sep 07 16:27:07.677943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:07.677909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k4rps"
Sep 07 16:27:07.961885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:07.961804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:08.245316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.245225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:08.430416 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.430382    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d5ccb59d-4xkqm"]
Sep 07 16:27:08.433887 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.433868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.437159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.437137    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 16:27:08.437284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.437173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhh4k\""
Sep 07 16:27:08.440346 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.440264    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d5ccb59d-4xkqm"]
Sep 07 16:27:08.482869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.482841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd0dd646-923f-4972-9e6b-58fd12e517d6-tls-cert\") pod \"authorino-d5ccb59d-4xkqm\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") " pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.483021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.482888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5dp\" (UniqueName: \"kubernetes.io/projected/fd0dd646-923f-4972-9e6b-58fd12e517d6-kube-api-access-dl5dp\") pod \"authorino-d5ccb59d-4xkqm\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") " pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.583746 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.583655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5dp\" (UniqueName: \"kubernetes.io/projected/fd0dd646-923f-4972-9e6b-58fd12e517d6-kube-api-access-dl5dp\") pod \"authorino-d5ccb59d-4xkqm\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") " pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.583944 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.583775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd0dd646-923f-4972-9e6b-58fd12e517d6-tls-cert\") pod \"authorino-d5ccb59d-4xkqm\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") " pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.586378 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.586357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd0dd646-923f-4972-9e6b-58fd12e517d6-tls-cert\") pod \"authorino-d5ccb59d-4xkqm\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") " pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.591192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.591173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5dp\" (UniqueName: \"kubernetes.io/projected/fd0dd646-923f-4972-9e6b-58fd12e517d6-kube-api-access-dl5dp\") pod \"authorino-d5ccb59d-4xkqm\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") " pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.744397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.744359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:27:08.988299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:08.988269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:09.010743 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:09.010717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d5ccb59d-4xkqm"]
Sep 07 16:27:09.013270 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:27:09.013233    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd0dd646_923f_4972_9e6b_58fd12e517d6.slice/crio-99799830863c9f1bbb61cd1bd949f89e711a33cda8eccc28716e1725d26ac495 WatchSource:0}: Error finding container 99799830863c9f1bbb61cd1bd949f89e711a33cda8eccc28716e1725d26ac495: Status 404 returned error can't find the container with id 99799830863c9f1bbb61cd1bd949f89e711a33cda8eccc28716e1725d26ac495
Sep 07 16:27:09.682742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:09.682699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5ccb59d-4xkqm" event={"ID":"fd0dd646-923f-4972-9e6b-58fd12e517d6","Type":"ContainerStarted","Data":"d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096"}
Sep 07 16:27:09.682742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:09.682748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5ccb59d-4xkqm" event={"ID":"fd0dd646-923f-4972-9e6b-58fd12e517d6","Type":"ContainerStarted","Data":"99799830863c9f1bbb61cd1bd949f89e711a33cda8eccc28716e1725d26ac495"}
Sep 07 16:27:09.698367 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:09.698316    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d5ccb59d-4xkqm" podStartSLOduration=1.167901156 podStartE2EDuration="1.698299446s" totalImagesPullingTime="530.39829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:27:09.696526015 +0000 UTC m=+434.578063008" watchObservedRunningTime="2026-09-07 16:27:09.698299446 +0000 UTC m=+434.579836413"
Sep 07 16:27:14.237016 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:14.236980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:14.337412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:14.337377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:14.391312 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:14.391286    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:14.456724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:14.456690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:17.748350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:17.748266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:17.861962 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:17.861932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:23.891959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:23.891922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:23.946352 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:23.946319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:50.376418 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:50.376379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:50.472676 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:50.472641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:51.708049 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:51.707993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:51.807762 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:51.807726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:51.908112 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:51.908077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:52.294979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:52.294941    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:27:52.404803 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:27:52.404771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:28:03.822386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:03.822352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:28:03.924485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:03.924448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:28:14.030260 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:14.030216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:28:14.088351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:14.088324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:28:22.429736 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.429702    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"]
Sep 07 16:28:22.433394 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.433374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.437573 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.437544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 07 16:28:22.437703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.437625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-stl6r\""
Sep 07 16:28:22.437790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.437633    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 16:28:22.440157 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.438348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 16:28:22.450503 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.450477    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"]
Sep 07 16:28:22.452818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.452789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:28:22.550071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.550037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.550250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.550089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.550250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.550194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.550250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.550229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a69e34-7ed4-4d99-a0fc-5874de5aa171-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.550366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.550265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.550366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.550281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w757q\" (UniqueName: \"kubernetes.io/projected/99a69e34-7ed4-4d99-a0fc-5874de5aa171-kube-api-access-w757q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a69e34-7ed4-4d99-a0fc-5874de5aa171-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w757q\" (UniqueName: \"kubernetes.io/projected/99a69e34-7ed4-4d99-a0fc-5874de5aa171-kube-api-access-w757q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651730 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.651922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.651900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.653490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.653466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a69e34-7ed4-4d99-a0fc-5874de5aa171-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.653724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.653701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a69e34-7ed4-4d99-a0fc-5874de5aa171-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.659062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.659039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w757q\" (UniqueName: \"kubernetes.io/projected/99a69e34-7ed4-4d99-a0fc-5874de5aa171-kube-api-access-w757q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2rll7\" (UID: \"99a69e34-7ed4-4d99-a0fc-5874de5aa171\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:22.750006 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:22.749921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:23.018449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:23.018426    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"]
Sep 07 16:28:23.021540 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:28:23.021513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a69e34_7ed4_4d99_a0fc_5874de5aa171.slice/crio-6241c070b644a6b9f0b47a82d9d3541f3bb57d5cb560e8fd35ee1b3e05c517fc WatchSource:0}: Error finding container 6241c070b644a6b9f0b47a82d9d3541f3bb57d5cb560e8fd35ee1b3e05c517fc: Status 404 returned error can't find the container with id 6241c070b644a6b9f0b47a82d9d3541f3bb57d5cb560e8fd35ee1b3e05c517fc
Sep 07 16:28:24.028673 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:24.028626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7" event={"ID":"99a69e34-7ed4-4d99-a0fc-5874de5aa171","Type":"ContainerStarted","Data":"6241c070b644a6b9f0b47a82d9d3541f3bb57d5cb560e8fd35ee1b3e05c517fc"}
Sep 07 16:28:25.035449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:25.035421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7" event={"ID":"99a69e34-7ed4-4d99-a0fc-5874de5aa171","Type":"ContainerStarted","Data":"807b5f57f6b2499cb9b7c8d0c6f258117054889904ab45eee03e70a94feb10cb"}
Sep 07 16:28:25.052821 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:25.052763    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7" podStartSLOduration=1.958945395 podStartE2EDuration="3.052734634s" totalImagesPullingTime="1.093789239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:28:25.05025986 +0000 UTC m=+509.931796839" watchObservedRunningTime="2026-09-07 16:28:25.052734634 +0000 UTC m=+509.934271601"
Sep 07 16:28:32.750423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:32.750382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:32.750800 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:32.750451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:32.762348 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:32.762326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:33.082702 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:33.082621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2rll7"
Sep 07 16:28:48.962354 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:28:48.962266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:29:04.517881 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:29:04.517833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:29:14.625317 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:29:14.625246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:29:23.855101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:29:23.855062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:29:37.470440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:29:37.470368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:29:45.238693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:29:45.238657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:00.136320 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.136284    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"]
Sep 07 16:30:00.139606 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.139591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"
Sep 07 16:30:00.142023 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.142003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:30:00.146847 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.146444    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"]
Sep 07 16:30:00.150022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.150002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"
Sep 07 16:30:00.418432 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.418407    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"]
Sep 07 16:30:00.420307 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:00.420275    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76c73ab_06b3_442b_bee7_446795081e41.slice/crio-406ee0493f8142b0b18e7dead95c41cafa2b51876c1a79502991f9d1a4cc6caf WatchSource:0}: Error finding container 406ee0493f8142b0b18e7dead95c41cafa2b51876c1a79502991f9d1a4cc6caf: Status 404 returned error can't find the container with id 406ee0493f8142b0b18e7dead95c41cafa2b51876c1a79502991f9d1a4cc6caf
Sep 07 16:30:00.479061 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.479018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47" event={"ID":"b76c73ab-06b3-442b-bee7-446795081e41","Type":"ContainerStarted","Data":"406ee0493f8142b0b18e7dead95c41cafa2b51876c1a79502991f9d1a4cc6caf"}
Sep 07 16:30:00.496554 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:00.496518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:03.494885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:03.494854    2577 generic.go:356] "Generic (PLEG): container finished" podID="b76c73ab-06b3-442b-bee7-446795081e41" containerID="f6f40ecc3ec5317dc6e7efb3279c4c920941aa957aaa979baae25f93309ca24a" exitCode=0
Sep 07 16:30:03.495253 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:03.494934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47" event={"ID":"b76c73ab-06b3-442b-bee7-446795081e41","Type":"ContainerDied","Data":"f6f40ecc3ec5317dc6e7efb3279c4c920941aa957aaa979baae25f93309ca24a"}
Sep 07 16:30:04.629489 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:04.629465    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"
Sep 07 16:30:05.504980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:05.504946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47" event={"ID":"b76c73ab-06b3-442b-bee7-446795081e41","Type":"ContainerDied","Data":"406ee0493f8142b0b18e7dead95c41cafa2b51876c1a79502991f9d1a4cc6caf"}
Sep 07 16:30:05.504980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:05.504979    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813310-65b47"
Sep 07 16:30:05.505192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:05.505019    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="406ee0493f8142b0b18e7dead95c41cafa2b51876c1a79502991f9d1a4cc6caf"
Sep 07 16:30:22.804796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:22.804705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:22.903198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:22.903166    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:23.592468 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.592438    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55cd58ff95-c565w"]
Sep 07 16:30:23.592811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.592799    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76c73ab-06b3-442b-bee7-446795081e41" containerName="cleanup"
Sep 07 16:30:23.592861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.592812    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76c73ab-06b3-442b-bee7-446795081e41" containerName="cleanup"
Sep 07 16:30:23.592904 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.592894    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76c73ab-06b3-442b-bee7-446795081e41" containerName="cleanup"
Sep 07 16:30:23.595241 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.595227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.608349 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.607973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55cd58ff95-c565w"]
Sep 07 16:30:23.650861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.650817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgx8\" (UniqueName: \"kubernetes.io/projected/6e8c7ee9-8c06-4f0b-8448-9c30c672ea66-kube-api-access-vdgx8\") pod \"authorino-55cd58ff95-c565w\" (UID: \"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66\") " pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.651046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.650903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e8c7ee9-8c06-4f0b-8448-9c30c672ea66-tls-cert\") pod \"authorino-55cd58ff95-c565w\" (UID: \"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66\") " pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.752446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.752396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgx8\" (UniqueName: \"kubernetes.io/projected/6e8c7ee9-8c06-4f0b-8448-9c30c672ea66-kube-api-access-vdgx8\") pod \"authorino-55cd58ff95-c565w\" (UID: \"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66\") " pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.752555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.752506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e8c7ee9-8c06-4f0b-8448-9c30c672ea66-tls-cert\") pod \"authorino-55cd58ff95-c565w\" (UID: \"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66\") " pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.755245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.755207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e8c7ee9-8c06-4f0b-8448-9c30c672ea66-tls-cert\") pod \"authorino-55cd58ff95-c565w\" (UID: \"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66\") " pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.760582 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.760559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgx8\" (UniqueName: \"kubernetes.io/projected/6e8c7ee9-8c06-4f0b-8448-9c30c672ea66-kube-api-access-vdgx8\") pod \"authorino-55cd58ff95-c565w\" (UID: \"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66\") " pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:23.904980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:23.904937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cd58ff95-c565w"
Sep 07 16:30:24.174379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:24.174319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55cd58ff95-c565w"]
Sep 07 16:30:24.176660 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:24.176636    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e8c7ee9_8c06_4f0b_8448_9c30c672ea66.slice/crio-b081927b0bcbf8a60026f91e84c725c373d254262b4a118c80c37a583d8aefdc WatchSource:0}: Error finding container b081927b0bcbf8a60026f91e84c725c373d254262b4a118c80c37a583d8aefdc: Status 404 returned error can't find the container with id b081927b0bcbf8a60026f91e84c725c373d254262b4a118c80c37a583d8aefdc
Sep 07 16:30:24.593240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:24.593161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cd58ff95-c565w" event={"ID":"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66","Type":"ContainerStarted","Data":"b081927b0bcbf8a60026f91e84c725c373d254262b4a118c80c37a583d8aefdc"}
Sep 07 16:30:25.599304 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.599269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cd58ff95-c565w" event={"ID":"6e8c7ee9-8c06-4f0b-8448-9c30c672ea66","Type":"ContainerStarted","Data":"3e008ec73d1335999499c88ee32f38a618d92befdf0bc486aa9be7c7a91b5bae"}
Sep 07 16:30:25.614546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.614492    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55cd58ff95-c565w" podStartSLOduration=1.996862892 podStartE2EDuration="2.614475975s" totalImagesPullingTime="617.613083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:25.612614146 +0000 UTC m=+630.494151139" watchObservedRunningTime="2026-09-07 16:30:25.614475975 +0000 UTC m=+630.496012942"
Sep 07 16:30:25.638207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.638175    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5ccb59d-4xkqm"]
Sep 07 16:30:25.638428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.638397    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d5ccb59d-4xkqm" podUID="fd0dd646-923f-4972-9e6b-58fd12e517d6" containerName="authorino" containerID="cri-o://d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096" gracePeriod=30
Sep 07 16:30:25.907567 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.907538    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:30:25.973950 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.973910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd0dd646-923f-4972-9e6b-58fd12e517d6-tls-cert\") pod \"fd0dd646-923f-4972-9e6b-58fd12e517d6\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") "
Sep 07 16:30:25.974157 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.974082    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl5dp\" (UniqueName: \"kubernetes.io/projected/fd0dd646-923f-4972-9e6b-58fd12e517d6-kube-api-access-dl5dp\") pod \"fd0dd646-923f-4972-9e6b-58fd12e517d6\" (UID: \"fd0dd646-923f-4972-9e6b-58fd12e517d6\") "
Sep 07 16:30:25.976518 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.976481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0dd646-923f-4972-9e6b-58fd12e517d6-kube-api-access-dl5dp" (OuterVolumeSpecName: "kube-api-access-dl5dp") pod "fd0dd646-923f-4972-9e6b-58fd12e517d6" (UID: "fd0dd646-923f-4972-9e6b-58fd12e517d6"). InnerVolumeSpecName "kube-api-access-dl5dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:30:25.985384 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:25.985350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd0dd646-923f-4972-9e6b-58fd12e517d6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fd0dd646-923f-4972-9e6b-58fd12e517d6" (UID: "fd0dd646-923f-4972-9e6b-58fd12e517d6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:30:26.075208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.075161    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl5dp\" (UniqueName: \"kubernetes.io/projected/fd0dd646-923f-4972-9e6b-58fd12e517d6-kube-api-access-dl5dp\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:30:26.075208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.075206    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd0dd646-923f-4972-9e6b-58fd12e517d6-tls-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:30:26.129416 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.129379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:26.236307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.236211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:26.605073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.604977    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd0dd646-923f-4972-9e6b-58fd12e517d6" containerID="d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096" exitCode=0
Sep 07 16:30:26.605073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.605064    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5ccb59d-4xkqm"
Sep 07 16:30:26.605579 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.605094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5ccb59d-4xkqm" event={"ID":"fd0dd646-923f-4972-9e6b-58fd12e517d6","Type":"ContainerDied","Data":"d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096"}
Sep 07 16:30:26.605579 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.605137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5ccb59d-4xkqm" event={"ID":"fd0dd646-923f-4972-9e6b-58fd12e517d6","Type":"ContainerDied","Data":"99799830863c9f1bbb61cd1bd949f89e711a33cda8eccc28716e1725d26ac495"}
Sep 07 16:30:26.605579 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.605157    2577 scope.go:117] "RemoveContainer" containerID="d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096"
Sep 07 16:30:26.613653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.613637    2577 scope.go:117] "RemoveContainer" containerID="d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096"
Sep 07 16:30:26.613949 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:30:26.613931    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096\": container with ID starting with d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096 not found: ID does not exist" containerID="d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096"
Sep 07 16:30:26.614029 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.613960    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096"} err="failed to get container status \"d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096\": rpc error: code = NotFound desc = could not find container \"d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096\": container with ID starting with d69c0c9250d1875ab1a9c7790cefad68bafe000da30b5c5d977fec82b4da2096 not found: ID does not exist"
Sep 07 16:30:26.630633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.630606    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5ccb59d-4xkqm"]
Sep 07 16:30:26.633082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:26.633059    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d5ccb59d-4xkqm"]
Sep 07 16:30:27.189833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:27.189800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:27.281646 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:27.281606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:27.567135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:27.567057    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd0dd646-923f-4972-9e6b-58fd12e517d6" path="/var/lib/kubelet/pods/fd0dd646-923f-4972-9e6b-58fd12e517d6/volumes"
Sep 07 16:30:42.379467 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.379435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:42.479005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.478966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:42.652178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.652145    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"]
Sep 07 16:30:42.652683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.652660    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0dd646-923f-4972-9e6b-58fd12e517d6" containerName="authorino"
Sep 07 16:30:42.652869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.652847    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0dd646-923f-4972-9e6b-58fd12e517d6" containerName="authorino"
Sep 07 16:30:42.653014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.653002    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0dd646-923f-4972-9e6b-58fd12e517d6" containerName="authorino"
Sep 07 16:30:42.660561 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.660530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.663009 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.662989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 16:30:42.663099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.663048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-747d6d-openshift-default-dockercfg-tl2pn\""
Sep 07 16:30:42.667120 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.667096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"]
Sep 07 16:30:42.682200 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.682170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:42.828325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istiod-ca-cert\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7kk5\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-kube-api-access-r7kk5\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-podinfo\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-data\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-certs\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-envoy\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-credential-socket\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-token\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.828724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.828674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-socket\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.871100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.871066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:42.930226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istiod-ca-cert\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7kk5\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-kube-api-access-r7kk5\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-podinfo\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-data\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-certs\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-envoy\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-credential-socket\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930504 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-token\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930741 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-socket\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930741 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-data\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-credential-socket\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.930879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.930789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-certs\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.931021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.931001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istiod-ca-cert\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.931079 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.931007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-socket\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.932885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.932847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-podinfo\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.932999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.932940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-envoy\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.937647 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.937621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7kk5\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-kube-api-access-r7kk5\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.938110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.938094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-token\") pod \"e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") " pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:42.973167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:42.973141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:43.025582 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:43.025319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:43.079346 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:43.079306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:43.251083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:43.251059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"]
Sep 07 16:30:43.253179 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:43.253151    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa2fc3f_9001_4f0b_8e0e_5aad52aba62f.slice/crio-327a972ab4b28cad6cf950f9b6814dbf88f6fe6cdc7c955890a8736d10d836bc WatchSource:0}: Error finding container 327a972ab4b28cad6cf950f9b6814dbf88f6fe6cdc7c955890a8736d10d836bc: Status 404 returned error can't find the container with id 327a972ab4b28cad6cf950f9b6814dbf88f6fe6cdc7c955890a8736d10d836bc
Sep 07 16:30:43.687814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:43.687783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" event={"ID":"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f","Type":"ContainerStarted","Data":"327a972ab4b28cad6cf950f9b6814dbf88f6fe6cdc7c955890a8736d10d836bc"}
Sep 07 16:30:43.738296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:43.738262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:43.836220 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:43.836194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:44.166734 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:44.166705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:45.263185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:45.263136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:46.118563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:46.118517    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:30:46.118713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:46.118595    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:30:46.118713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:46.118623    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:30:46.707298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:46.707265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" event={"ID":"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f","Type":"ContainerStarted","Data":"211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"}
Sep 07 16:30:46.725906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:46.725857    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podStartSLOduration=1.8629663810000001 podStartE2EDuration="4.725842506s" totalImagesPullingTime="2.862876125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:46.72460082 +0000 UTC m=+651.606137813" watchObservedRunningTime="2026-09-07 16:30:46.725842506 +0000 UTC m=+651.607379473"
Sep 07 16:30:46.973947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:46.973856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:47.978916 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:47.978884    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:48.580939 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:48.580906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:48.679331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:48.679300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:48.716842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:48.716814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:48.717864 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:48.717841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:30:49.865311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:49.865272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:49.965086 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:49.965054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:50.064617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.064588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:50.407513 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.407482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:50.505490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.505459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:50.547079 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.546974    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"]
Sep 07 16:30:50.550685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.550658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.553397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.553375    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-747d6d-dockercfg-28n7g\""
Sep 07 16:30:50.553642 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.553626    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-747d6d\""
Sep 07 16:30:50.561816 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.561794    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"]
Sep 07 16:30:50.595421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.595381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqgb\" (UniqueName: \"kubernetes.io/projected/5ce05475-9d94-4808-8cc0-6b462004bf57-kube-api-access-nrqgb\") pod \"payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.595588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.595452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce05475-9d94-4808-8cc0-6b462004bf57-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.603938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.603894    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:50.696414 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.696328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqgb\" (UniqueName: \"kubernetes.io/projected/5ce05475-9d94-4808-8cc0-6b462004bf57-kube-api-access-nrqgb\") pod \"payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.696414 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.696389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce05475-9d94-4808-8cc0-6b462004bf57-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.697119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.697100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce05475-9d94-4808-8cc0-6b462004bf57-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.704584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.704553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqgb\" (UniqueName: \"kubernetes.io/projected/5ce05475-9d94-4808-8cc0-6b462004bf57-kube-api-access-nrqgb\") pod \"payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.861714 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.861676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:50.877273 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:50.877228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:51.132925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:51.132898    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"]
Sep 07 16:30:51.135333 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:51.135297    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce05475_9d94_4808_8cc0_6b462004bf57.slice/crio-a769911f789b73343c66d104a0eb8bb554cb0a14ad6bb9e1bde411c338750ef6 WatchSource:0}: Error finding container a769911f789b73343c66d104a0eb8bb554cb0a14ad6bb9e1bde411c338750ef6: Status 404 returned error can't find the container with id a769911f789b73343c66d104a0eb8bb554cb0a14ad6bb9e1bde411c338750ef6
Sep 07 16:30:51.515000 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:51.514962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:51.615364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:51.615312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:51.715287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:51.715256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:51.734406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:51.734375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8" event={"ID":"5ce05475-9d94-4808-8cc0-6b462004bf57","Type":"ContainerStarted","Data":"a769911f789b73343c66d104a0eb8bb554cb0a14ad6bb9e1bde411c338750ef6"}
Sep 07 16:30:51.815320 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:51.815246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:52.087622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:52.087524    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:52.407403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:52.407365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:53.746003 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:53.745972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8" event={"ID":"5ce05475-9d94-4808-8cc0-6b462004bf57","Type":"ContainerStarted","Data":"049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4"}
Sep 07 16:30:53.746420 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:53.746103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:30:53.763386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:53.763340    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8" podStartSLOduration=2.098770889 podStartE2EDuration="3.763326146s" totalImagesPullingTime="1.664555257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:53.761955237 +0000 UTC m=+658.643492242" watchObservedRunningTime="2026-09-07 16:30:53.763326146 +0000 UTC m=+658.644863114"
Sep 07 16:30:54.144667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.144636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:54.223493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.223458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:54.223995 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.223973    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"]
Sep 07 16:30:54.227974 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.227952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.230539 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.230518    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-e4ed13ae-kserve-self-signed-certs\""
Sep 07 16:30:54.236901 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.236879    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"]
Sep 07 16:30:54.330927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.330891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-tmp-dir\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.331103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.330934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-home\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.331103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.330951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-model-cache\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.331103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.331032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34115e-4a33-426a-8fed-63984218bab2-tls-certs\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.331103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.331054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-dshm\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.331103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.331082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtcq\" (UniqueName: \"kubernetes.io/projected/ef34115e-4a33-426a-8fed-63984218bab2-kube-api-access-7gtcq\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.431892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.431806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34115e-4a33-426a-8fed-63984218bab2-tls-certs\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.431892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.431844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-dshm\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.431892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.431867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtcq\" (UniqueName: \"kubernetes.io/projected/ef34115e-4a33-426a-8fed-63984218bab2-kube-api-access-7gtcq\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.432172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.431923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-tmp-dir\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.432172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.431942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-home\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.432172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.431958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-model-cache\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.432388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.432368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-model-cache\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.432388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.432382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-tmp-dir\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.432492 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.432405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-home\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.434379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.434356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-dshm\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.434517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.434503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34115e-4a33-426a-8fed-63984218bab2-tls-certs\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.439510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.439492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtcq\" (UniqueName: \"kubernetes.io/projected/ef34115e-4a33-426a-8fed-63984218bab2-kube-api-access-7gtcq\") pod \"e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") " pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.538187 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.538140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:30:54.545630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.545595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:54.645186 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.645145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:54.749710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.749622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:54.826854 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:54.826815    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"]
Sep 07 16:30:55.320240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.320164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:55.445025 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.444996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:55.538234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.538202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:55.759876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.759846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps" event={"ID":"ef34115e-4a33-426a-8fed-63984218bab2","Type":"ContainerStarted","Data":"72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50"}
Sep 07 16:30:55.760328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.759891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps" event={"ID":"ef34115e-4a33-426a-8fed-63984218bab2","Type":"ContainerStarted","Data":"390ea7eec98d8af8f9c74db4bf9fb3715624dd35a0ed64d5d40051ac77d8e484"}
Sep 07 16:30:55.777141 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.777081    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps" podStartSLOduration=1.777060904 podStartE2EDuration="1.777060904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:55.776001485 +0000 UTC m=+660.657538455" watchObservedRunningTime="2026-09-07 16:30:55.777060904 +0000 UTC m=+660.658597872"
Sep 07 16:30:55.798298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.798261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:55.874527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.874495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:55.897173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.897130    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"]
Sep 07 16:30:55.902041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.902021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:55.904475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.904450    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-8a3403-dockercfg-b644f\""
Sep 07 16:30:55.904604 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.904454    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8a3403\""
Sep 07 16:30:55.935779 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.930806    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"]
Sep 07 16:30:55.946506 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.946468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:55.946648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:55.946549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626g7\" (UniqueName: \"kubernetes.io/projected/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-kube-api-access-626g7\") pod \"payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.047229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.047149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.047229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.047188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626g7\" (UniqueName: \"kubernetes.io/projected/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-kube-api-access-626g7\") pod \"payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.047848 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.047831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.055471 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.055447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-626g7\" (UniqueName: \"kubernetes.io/projected/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-kube-api-access-626g7\") pod \"payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.174473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.174433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:56.226777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.225923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.278845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.278775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:56.377317 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.377223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:56.498309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.498284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"]
Sep 07 16:30:56.499689 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:56.499662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d85d3c_b7a2_40c5_99a0_9b2166bac792.slice/crio-d001d0600b04f79ce504320d615dca4daa4ffbd6dc8c4f14ece1e5eb465ba4f4 WatchSource:0}: Error finding container d001d0600b04f79ce504320d615dca4daa4ffbd6dc8c4f14ece1e5eb465ba4f4: Status 404 returned error can't find the container with id d001d0600b04f79ce504320d615dca4daa4ffbd6dc8c4f14ece1e5eb465ba4f4
Sep 07 16:30:56.637550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.637523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:56.726517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.726047    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"]
Sep 07 16:30:56.726517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.726444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:56.730353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.730331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.736324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.736275    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-983d9f00\""
Sep 07 16:30:56.738100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.738017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"]
Sep 07 16:30:56.754162 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.754138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f614673-3318-4966-912c-75bfb9ba68ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.754261 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.754220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcvg\" (UniqueName: \"kubernetes.io/projected/0f614673-3318-4966-912c-75bfb9ba68ab-kube-api-access-qfcvg\") pod \"payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.766161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.766134    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"]
Sep 07 16:30:56.769854 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.769835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm" event={"ID":"a2d85d3c-b7a2-40c5-99a0-9b2166bac792","Type":"ContainerStarted","Data":"e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d"}
Sep 07 16:30:56.769941 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.769861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm" event={"ID":"a2d85d3c-b7a2-40c5-99a0-9b2166bac792","Type":"ContainerStarted","Data":"d001d0600b04f79ce504320d615dca4daa4ffbd6dc8c4f14ece1e5eb465ba4f4"}
Sep 07 16:30:56.769941 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.769887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:30:56.769941 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.769931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:56.772095 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.772080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-983d9f00-dockercfg-jjbb5\""
Sep 07 16:30:56.781258 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.781242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"]
Sep 07 16:30:56.790478 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.790430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm" podStartSLOduration=1.790418069 podStartE2EDuration="1.790418069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:56.789548619 +0000 UTC m=+661.671085588" watchObservedRunningTime="2026-09-07 16:30:56.790418069 +0000 UTC m=+661.671955035"
Sep 07 16:30:56.855334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.855302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcvg\" (UniqueName: \"kubernetes.io/projected/0f614673-3318-4966-912c-75bfb9ba68ab-kube-api-access-qfcvg\") pod \"payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.855513 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.855342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2b8c28-413c-440e-bf47-82161e111168-plugins-config-volume\") pod \"payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") " pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:56.855513 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.855399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzzf\" (UniqueName: \"kubernetes.io/projected/3b2b8c28-413c-440e-bf47-82161e111168-kube-api-access-xkzzf\") pod \"payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") " pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:56.855995 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.855638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f614673-3318-4966-912c-75bfb9ba68ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.856275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.856246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f614673-3318-4966-912c-75bfb9ba68ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.865806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.865785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcvg\" (UniqueName: \"kubernetes.io/projected/0f614673-3318-4966-912c-75bfb9ba68ab-kube-api-access-qfcvg\") pod \"payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:56.957062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.956984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzzf\" (UniqueName: \"kubernetes.io/projected/3b2b8c28-413c-440e-bf47-82161e111168-kube-api-access-xkzzf\") pod \"payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") " pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:56.957197 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.957084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2b8c28-413c-440e-bf47-82161e111168-plugins-config-volume\") pod \"payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") " pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:56.957687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.957665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2b8c28-413c-440e-bf47-82161e111168-plugins-config-volume\") pod \"payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") " pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:56.970836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:56.970816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzzf\" (UniqueName: \"kubernetes.io/projected/3b2b8c28-413c-440e-bf47-82161e111168-kube-api-access-xkzzf\") pod \"payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") " pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:57.045889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.045861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:57.080164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.079978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:57.139779 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.136207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:57.321828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.319666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"]
Sep 07 16:30:57.335779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:57.335175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f614673_3318_4966_912c_75bfb9ba68ab.slice/crio-e4157537f1d6a6accff09a3bd223c45bac04096e8fcbb9ba71ba2d3e12c0ae94 WatchSource:0}: Error finding container e4157537f1d6a6accff09a3bd223c45bac04096e8fcbb9ba71ba2d3e12c0ae94: Status 404 returned error can't find the container with id e4157537f1d6a6accff09a3bd223c45bac04096e8fcbb9ba71ba2d3e12c0ae94
Sep 07 16:30:57.392307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.392287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"]
Sep 07 16:30:57.393521 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:30:57.393499    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2b8c28_413c_440e_bf47_82161e111168.slice/crio-7217ba5bab66d021f95365b0e9a227cb3c35a662f61356dfdd4457e41bfad8a3 WatchSource:0}: Error finding container 7217ba5bab66d021f95365b0e9a227cb3c35a662f61356dfdd4457e41bfad8a3: Status 404 returned error can't find the container with id 7217ba5bab66d021f95365b0e9a227cb3c35a662f61356dfdd4457e41bfad8a3
Sep 07 16:30:57.772437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.772403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:30:57.775108 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.775079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd" event={"ID":"3b2b8c28-413c-440e-bf47-82161e111168","Type":"ContainerStarted","Data":"50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d"}
Sep 07 16:30:57.775227 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.775130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd" event={"ID":"3b2b8c28-413c-440e-bf47-82161e111168","Type":"ContainerStarted","Data":"7217ba5bab66d021f95365b0e9a227cb3c35a662f61356dfdd4457e41bfad8a3"}
Sep 07 16:30:57.775227 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.775195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:30:57.776869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.776841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt" event={"ID":"0f614673-3318-4966-912c-75bfb9ba68ab","Type":"ContainerStarted","Data":"e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01"}
Sep 07 16:30:57.776989 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.776889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt" event={"ID":"0f614673-3318-4966-912c-75bfb9ba68ab","Type":"ContainerStarted","Data":"e4157537f1d6a6accff09a3bd223c45bac04096e8fcbb9ba71ba2d3e12c0ae94"}
Sep 07 16:30:57.776989 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.776957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:30:57.792932 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.792881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd" podStartSLOduration=1.792867451 podStartE2EDuration="1.792867451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:57.790990901 +0000 UTC m=+662.672527871" watchObservedRunningTime="2026-09-07 16:30:57.792867451 +0000 UTC m=+662.674404471"
Sep 07 16:30:57.808614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:30:57.808560    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt" podStartSLOduration=1.808541653 podStartE2EDuration="1.808541653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:57.807196487 +0000 UTC m=+662.688733458" watchObservedRunningTime="2026-09-07 16:30:57.808541653 +0000 UTC m=+662.690078626"
Sep 07 16:31:00.982876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:00.982834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:01.126993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:01.126922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:01.241045 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:01.240952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:01.336651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:01.336623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:03.718069 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.718032    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:03.718505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.718095    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:03.810185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.810156    2577 generic.go:356] "Generic (PLEG): container finished" podID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerID="211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21" exitCode=0
Sep 07 16:31:03.810370 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.810235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" event={"ID":"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f","Type":"ContainerDied","Data":"211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"}
Sep 07 16:31:03.810671 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.810657    2577 scope.go:117] "RemoveContainer" containerID="211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"
Sep 07 16:31:03.811768 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.811724    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:03.811869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.811820    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:03.811931 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.811907    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:03.960406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:03.960374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:04.541588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.541552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:31:04.543160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.543129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:31:04.557895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.557865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:31:04.753226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.753202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:31:04.838461 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.838373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" event={"ID":"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f","Type":"ContainerStarted","Data":"2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1"}
Sep 07 16:31:04.850221 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.850191    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:31:04.974197 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.974171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:31:04.974590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.974566    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:04.974776 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:04.974734    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:05.472207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:05.472163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:05.974531 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:05.974487    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:05.974961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:05.974552    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:06.973593 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:06.973552    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:06.973821 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:06.973609    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:07.779029 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:07.778993    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:31:07.973653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:07.973616    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:07.973856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:07.973669    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:08.783963 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:08.783933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:31:08.784348 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:08.784001    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:31:08.974524 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:08.974486    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:08.974691 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:08.974548    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:09.974085 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:09.974039    2577 patch_prober.go:28] interesting pod/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 16:31:09.974549 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:09.974102    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 16:31:10.778510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:10.778475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:10.978625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:10.978592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:31:10.979050 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:10.979015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:31:10.980002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:10.979983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:31:13.490535 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:13.490499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:13.794597 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:13.794506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:14.091325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.091247    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"]
Sep 07 16:31:14.099715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.099694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.102370 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.102297    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a7d399-openshift-default-dockercfg-b4pxw\""
Sep 07 16:31:14.105265 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.105243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"]
Sep 07 16:31:14.116471 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.116447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:14.231418 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb82f\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-kube-api-access-qb82f\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231418 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-certs\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-socket\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-istio-token\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1fab42-95c4-4eff-b873-0beba327abea-istiod-ca-cert\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-data\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-credential-socket\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-envoy\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.231843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.231732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1fab42-95c4-4eff-b873-0beba327abea-istio-podinfo\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.249565 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.249532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:14.333021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.332985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-istio-token\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1fab42-95c4-4eff-b873-0beba327abea-istiod-ca-cert\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-data\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-credential-socket\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-envoy\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1fab42-95c4-4eff-b873-0beba327abea-istio-podinfo\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb82f\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-kube-api-access-qb82f\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-certs\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-socket\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-credential-socket\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333637 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-data\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333831 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-certs\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333910 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1fab42-95c4-4eff-b873-0beba327abea-istiod-ca-cert\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.333910 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.333869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-socket\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.335703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.335678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-envoy\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.335893 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.335873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1fab42-95c4-4eff-b873-0beba327abea-istio-podinfo\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.341379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.341317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb82f\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-kube-api-access-qb82f\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.341667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.341645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-istio-token\") pod \"e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") " pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.410985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.410946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:14.685208 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.685187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"]
Sep 07 16:31:14.687049 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:31:14.687022    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1fab42_95c4_4eff_b873_0beba327abea.slice/crio-e38ef293d48365fc079a47fa9febe74c75010d5326286ace60a20ef2e806a3ea WatchSource:0}: Error finding container e38ef293d48365fc079a47fa9febe74c75010d5326286ace60a20ef2e806a3ea: Status 404 returned error can't find the container with id e38ef293d48365fc079a47fa9febe74c75010d5326286ace60a20ef2e806a3ea
Sep 07 16:31:14.689371 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.689332    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:14.689469 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.689400    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:14.689469 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.689437    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:14.901650 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.901613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" event={"ID":"3d1fab42-95c4-4eff-b873-0beba327abea","Type":"ContainerStarted","Data":"2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"}
Sep 07 16:31:14.901881 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.901680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" event={"ID":"3d1fab42-95c4-4eff-b873-0beba327abea","Type":"ContainerStarted","Data":"e38ef293d48365fc079a47fa9febe74c75010d5326286ace60a20ef2e806a3ea"}
Sep 07 16:31:14.923072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:14.923018    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podStartSLOduration=0.923001594 podStartE2EDuration="923.001594ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:14.921297678 +0000 UTC m=+679.802834647" watchObservedRunningTime="2026-09-07 16:31:14.923001594 +0000 UTC m=+679.804538564"
Sep 07 16:31:15.411150 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:15.411118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:15.415970 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:15.415945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:15.907071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:15.907048    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:15.908094 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:15.908075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:16.162681 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:16.162593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:16.749825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:16.749738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:16.852219 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:16.852191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:16.957282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:16.957245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:17.465192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:17.465138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:18.201993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:18.201966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:18.933790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:18.933731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:19.030495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.030464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:19.415352 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.415327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:19.515605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.515565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:19.531229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.531198    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"]
Sep 07 16:31:19.534960 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.534939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.539490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.539462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a7d399-dockercfg-nrltd\""
Sep 07 16:31:19.539746 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.539731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a7d399\""
Sep 07 16:31:19.548415 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.548388    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"]
Sep 07 16:31:19.550731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.550708    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"]
Sep 07 16:31:19.554478 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.554460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.571498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.571467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"]
Sep 07 16:31:19.586798 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.586740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:19.698570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.698493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.698728 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.698640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnr5k\" (UniqueName: \"kubernetes.io/projected/021a428a-b211-454b-b1ef-0235e1b939f2-kube-api-access-vnr5k\") pod \"payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.698728 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.698693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zxw\" (UniqueName: \"kubernetes.io/projected/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-kube-api-access-k8zxw\") pod \"payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.705913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.700162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021a428a-b211-454b-b1ef-0235e1b939f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.800984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.800948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnr5k\" (UniqueName: \"kubernetes.io/projected/021a428a-b211-454b-b1ef-0235e1b939f2-kube-api-access-vnr5k\") pod \"payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.800984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.800990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zxw\" (UniqueName: \"kubernetes.io/projected/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-kube-api-access-k8zxw\") pod \"payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.801309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.801055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021a428a-b211-454b-b1ef-0235e1b939f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.801309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.801124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.801932 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.801908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021a428a-b211-454b-b1ef-0235e1b939f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.802018 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.801935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.815289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.815266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zxw\" (UniqueName: \"kubernetes.io/projected/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-kube-api-access-k8zxw\") pod \"payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.822981 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.822955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnr5k\" (UniqueName: \"kubernetes.io/projected/021a428a-b211-454b-b1ef-0235e1b939f2-kube-api-access-vnr5k\") pod \"payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.846600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.846563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:19.867353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.867330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:19.867823 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.867788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:19.955629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:19.955413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:20.055378 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.055343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:20.121106 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.121076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"]
Sep 07 16:31:20.122728 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:31:20.122701    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287cc9a7_7bc1_45d6_b9d1_869761a7c63f.slice/crio-229aed66c0a4bb0abda5b3b70c20b8cb30f7eb22ee13b8dccd511388c65986e9 WatchSource:0}: Error finding container 229aed66c0a4bb0abda5b3b70c20b8cb30f7eb22ee13b8dccd511388c65986e9: Status 404 returned error can't find the container with id 229aed66c0a4bb0abda5b3b70c20b8cb30f7eb22ee13b8dccd511388c65986e9
Sep 07 16:31:20.127268 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.127244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:20.165247 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.165225    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"]
Sep 07 16:31:20.166818 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:31:20.166789    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021a428a_b211_454b_b1ef_0235e1b939f2.slice/crio-5b4489cec29dd96478555b085a530e3efb9f8631128f0e0f05adc5cc708b228b WatchSource:0}: Error finding container 5b4489cec29dd96478555b085a530e3efb9f8631128f0e0f05adc5cc708b228b: Status 404 returned error can't find the container with id 5b4489cec29dd96478555b085a530e3efb9f8631128f0e0f05adc5cc708b228b
Sep 07 16:31:20.862837 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.862798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:20.935694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.935655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q" event={"ID":"287cc9a7-7bc1-45d6-b9d1-869761a7c63f","Type":"ContainerStarted","Data":"135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854"}
Sep 07 16:31:20.935943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.935721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q" event={"ID":"287cc9a7-7bc1-45d6-b9d1-869761a7c63f","Type":"ContainerStarted","Data":"229aed66c0a4bb0abda5b3b70c20b8cb30f7eb22ee13b8dccd511388c65986e9"}
Sep 07 16:31:20.935943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.935788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:20.937852 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.937819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr" event={"ID":"021a428a-b211-454b-b1ef-0235e1b939f2","Type":"ContainerStarted","Data":"fb7a06a7d0961cfb449a46888c5f4a183484662b0c1941a3936ec0237ad85ffd"}
Sep 07 16:31:20.938004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.937881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr" event={"ID":"021a428a-b211-454b-b1ef-0235e1b939f2","Type":"ContainerStarted","Data":"5b4489cec29dd96478555b085a530e3efb9f8631128f0e0f05adc5cc708b228b"}
Sep 07 16:31:20.938004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.937990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:20.950780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.950704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q" podStartSLOduration=1.950684969 podStartE2EDuration="1.950684969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:20.950341822 +0000 UTC m=+685.831878803" watchObservedRunningTime="2026-09-07 16:31:20.950684969 +0000 UTC m=+685.832221937"
Sep 07 16:31:20.967507 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:20.967450    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr" podStartSLOduration=1.967431631 podStartE2EDuration="1.967431631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:20.965880322 +0000 UTC m=+685.847417291" watchObservedRunningTime="2026-09-07 16:31:20.967431631 +0000 UTC m=+685.848968625"
Sep 07 16:31:21.771182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:21.771151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:25.845557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:25.845508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:25.951430 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:25.951386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:26.047232 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:26.047200    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:26.160812 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:26.160783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:31.774595 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:31.774566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:31.829126 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:31.829086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:31.945354 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:31.945325    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:31:31.945534 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:31.945393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:31:36.180131 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.180044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:36.203864 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.203827    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"]
Sep 07 16:31:36.204154 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.204098    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd" podUID="3b2b8c28-413c-440e-bf47-82161e111168" containerName="payload-processing" containerID="cri-o://50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d" gracePeriod=30
Sep 07 16:31:36.210588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.210560    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"]
Sep 07 16:31:36.210830 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.210808    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt" podUID="0f614673-3318-4966-912c-75bfb9ba68ab" containerName="payload-pre-processing" containerID="cri-o://e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01" gracePeriod=30
Sep 07 16:31:36.248380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.248345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:36.343424 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.343350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:36.443164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.442896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:36.466977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.466203    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:31:36.504681 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.504581    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:31:36.508689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.508656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:36.578490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.578460    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f614673-3318-4966-912c-75bfb9ba68ab-plugins-config-volume\") pod \"0f614673-3318-4966-912c-75bfb9ba68ab\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") "
Sep 07 16:31:36.578490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.578496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2b8c28-413c-440e-bf47-82161e111168-plugins-config-volume\") pod \"3b2b8c28-413c-440e-bf47-82161e111168\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") "
Sep 07 16:31:36.578781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.578588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkzzf\" (UniqueName: \"kubernetes.io/projected/3b2b8c28-413c-440e-bf47-82161e111168-kube-api-access-xkzzf\") pod \"3b2b8c28-413c-440e-bf47-82161e111168\" (UID: \"3b2b8c28-413c-440e-bf47-82161e111168\") "
Sep 07 16:31:36.578781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.578622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfcvg\" (UniqueName: \"kubernetes.io/projected/0f614673-3318-4966-912c-75bfb9ba68ab-kube-api-access-qfcvg\") pod \"0f614673-3318-4966-912c-75bfb9ba68ab\" (UID: \"0f614673-3318-4966-912c-75bfb9ba68ab\") "
Sep 07 16:31:36.578898 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.578879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f614673-3318-4966-912c-75bfb9ba68ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f614673-3318-4966-912c-75bfb9ba68ab" (UID: "0f614673-3318-4966-912c-75bfb9ba68ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:31:36.579038 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.579018    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f614673-3318-4966-912c-75bfb9ba68ab-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:31:36.579186 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.579169    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2b8c28-413c-440e-bf47-82161e111168-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b2b8c28-413c-440e-bf47-82161e111168" (UID: "3b2b8c28-413c-440e-bf47-82161e111168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:31:36.582054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.581428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2b8c28-413c-440e-bf47-82161e111168-kube-api-access-xkzzf" (OuterVolumeSpecName: "kube-api-access-xkzzf") pod "3b2b8c28-413c-440e-bf47-82161e111168" (UID: "3b2b8c28-413c-440e-bf47-82161e111168"). InnerVolumeSpecName "kube-api-access-xkzzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:31:36.582054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.581824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f614673-3318-4966-912c-75bfb9ba68ab-kube-api-access-qfcvg" (OuterVolumeSpecName: "kube-api-access-qfcvg") pod "0f614673-3318-4966-912c-75bfb9ba68ab" (UID: "0f614673-3318-4966-912c-75bfb9ba68ab"). InnerVolumeSpecName "kube-api-access-qfcvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:31:36.679885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.679852    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2b8c28-413c-440e-bf47-82161e111168-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:31:36.679885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.679878    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzzf\" (UniqueName: \"kubernetes.io/projected/3b2b8c28-413c-440e-bf47-82161e111168-kube-api-access-xkzzf\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:31:36.679885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.679887    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcvg\" (UniqueName: \"kubernetes.io/projected/0f614673-3318-4966-912c-75bfb9ba68ab-kube-api-access-qfcvg\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:31:36.954996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:36.954956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:37.025181 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.025140    2577 generic.go:356] "Generic (PLEG): container finished" podID="3b2b8c28-413c-440e-bf47-82161e111168" containerID="50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d" exitCode=0
Sep 07 16:31:37.025363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.025193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"
Sep 07 16:31:37.025363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.025228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd" event={"ID":"3b2b8c28-413c-440e-bf47-82161e111168","Type":"ContainerDied","Data":"50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d"}
Sep 07 16:31:37.025363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.025288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd" event={"ID":"3b2b8c28-413c-440e-bf47-82161e111168","Type":"ContainerDied","Data":"7217ba5bab66d021f95365b0e9a227cb3c35a662f61356dfdd4457e41bfad8a3"}
Sep 07 16:31:37.025363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.025318    2577 scope.go:117] "RemoveContainer" containerID="50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d"
Sep 07 16:31:37.026844 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.026820    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f614673-3318-4966-912c-75bfb9ba68ab" containerID="e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01" exitCode=0
Sep 07 16:31:37.026967 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.026870    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"
Sep 07 16:31:37.026967 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.026929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt" event={"ID":"0f614673-3318-4966-912c-75bfb9ba68ab","Type":"ContainerDied","Data":"e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01"}
Sep 07 16:31:37.027070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.026974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt" event={"ID":"0f614673-3318-4966-912c-75bfb9ba68ab","Type":"ContainerDied","Data":"e4157537f1d6a6accff09a3bd223c45bac04096e8fcbb9ba71ba2d3e12c0ae94"}
Sep 07 16:31:37.036669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.036653    2577 scope.go:117] "RemoveContainer" containerID="50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d"
Sep 07 16:31:37.037076 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:31:37.037054    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d\": container with ID starting with 50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d not found: ID does not exist" containerID="50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d"
Sep 07 16:31:37.037166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.037083    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d"} err="failed to get container status \"50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d\": rpc error: code = NotFound desc = could not find container \"50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d\": container with ID starting with 50037ec4c62e5cc96e31595c681d817c0a708ac3e94b8c4c0e26015f915c562d not found: ID does not exist"
Sep 07 16:31:37.037166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.037103    2577 scope.go:117] "RemoveContainer" containerID="e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01"
Sep 07 16:31:37.046485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.046468    2577 scope.go:117] "RemoveContainer" containerID="e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01"
Sep 07 16:31:37.046806 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:31:37.046746    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01\": container with ID starting with e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01 not found: ID does not exist" containerID="e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01"
Sep 07 16:31:37.046872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.046812    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01"} err="failed to get container status \"e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01\": rpc error: code = NotFound desc = could not find container \"e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01\": container with ID starting with e14a4fedc1998cdfc37a55e6ef42189a1a9d2aa3e8a26b6fccce3a127bac7f01 not found: ID does not exist"
Sep 07 16:31:37.058218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.058190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:37.058598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.058579    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"]
Sep 07 16:31:37.064770 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.064733    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-983d9f00-6c5bd6bbf8-77nlt"]
Sep 07 16:31:37.080299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.080252    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"]
Sep 07 16:31:37.083376 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.083351    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-983d9f00-6b8d99cc89-7nndd"]
Sep 07 16:31:37.153132 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.153096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:37.424253 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.424221    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"]
Sep 07 16:31:37.424939 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.424910    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2b8c28-413c-440e-bf47-82161e111168" containerName="payload-processing"
Sep 07 16:31:37.424939 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.424944    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2b8c28-413c-440e-bf47-82161e111168" containerName="payload-processing"
Sep 07 16:31:37.425173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.424985    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f614673-3318-4966-912c-75bfb9ba68ab" containerName="payload-pre-processing"
Sep 07 16:31:37.425173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.424994    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f614673-3318-4966-912c-75bfb9ba68ab" containerName="payload-pre-processing"
Sep 07 16:31:37.425173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.425111    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2b8c28-413c-440e-bf47-82161e111168" containerName="payload-processing"
Sep 07 16:31:37.425173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.425125    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f614673-3318-4966-912c-75bfb9ba68ab" containerName="payload-pre-processing"
Sep 07 16:31:37.430182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.430160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.432572 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.432552    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1e4cf2b6-gw-openshift-default-dockercfg-cjs27\""
Sep 07 16:31:37.438284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.438263    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"]
Sep 07 16:31:37.458306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.458280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:37.487550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-certs\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj85z\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-kube-api-access-gj85z\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-credential-socket\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487737 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-envoy\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487737 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985bac87-b1cf-47a5-964f-bf983c850c96-istiod-ca-cert\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487737 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985bac87-b1cf-47a5-964f-bf983c850c96-istio-podinfo\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487737 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-istio-token\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-socket\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.487909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.487805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-data\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.576634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.576603    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f614673-3318-4966-912c-75bfb9ba68ab" path="/var/lib/kubelet/pods/0f614673-3318-4966-912c-75bfb9ba68ab/volumes"
Sep 07 16:31:37.577048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.577033    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2b8c28-413c-440e-bf47-82161e111168" path="/var/lib/kubelet/pods/3b2b8c28-413c-440e-bf47-82161e111168/volumes"
Sep 07 16:31:37.588996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.588975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-data\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-certs\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj85z\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-kube-api-access-gj85z\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-credential-socket\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-envoy\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985bac87-b1cf-47a5-964f-bf983c850c96-istiod-ca-cert\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985bac87-b1cf-47a5-964f-bf983c850c96-istio-podinfo\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-istio-token\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-socket\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-data\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589571 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-certs\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-credential-socket\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985bac87-b1cf-47a5-964f-bf983c850c96-istiod-ca-cert\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.589954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.589918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-socket\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.591785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.591736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-envoy\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.591868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.591827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985bac87-b1cf-47a5-964f-bf983c850c96-istio-podinfo\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.600810 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.600788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-istio-token\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.601147 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.601126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj85z\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-kube-api-access-gj85z\") pod \"e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") " pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.742991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.742908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:37.902209 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:37.902174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:38.020967 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:38.020939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"]
Sep 07 16:31:38.022836 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:31:38.022810    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985bac87_b1cf_47a5_964f_bf983c850c96.slice/crio-e199e7864564e9565b095dce6f63a0135480acc02ae687909d5fdad6a5a68fb6 WatchSource:0}: Error finding container e199e7864564e9565b095dce6f63a0135480acc02ae687909d5fdad6a5a68fb6: Status 404 returned error can't find the container with id e199e7864564e9565b095dce6f63a0135480acc02ae687909d5fdad6a5a68fb6
Sep 07 16:31:38.025222 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:38.025190    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:38.025296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:38.025259    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:38.025296 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:38.025289    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:38.035497 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:38.035471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" event={"ID":"985bac87-b1cf-47a5-964f-bf983c850c96","Type":"ContainerStarted","Data":"e199e7864564e9565b095dce6f63a0135480acc02ae687909d5fdad6a5a68fb6"}
Sep 07 16:31:39.043838 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:39.043805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" event={"ID":"985bac87-b1cf-47a5-964f-bf983c850c96","Type":"ContainerStarted","Data":"fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"}
Sep 07 16:31:39.065322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:39.065275    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podStartSLOduration=2.065256948 podStartE2EDuration="2.065256948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:39.061942179 +0000 UTC m=+703.943479158" watchObservedRunningTime="2026-09-07 16:31:39.065256948 +0000 UTC m=+703.946794282"
Sep 07 16:31:39.743522 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:39.743484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:39.749073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:39.749049    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:39.859032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:39.859003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:40.050713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.050635    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d1fab42-95c4-4eff-b873-0beba327abea" containerID="2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160" exitCode=0
Sep 07 16:31:40.051084 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.050709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" event={"ID":"3d1fab42-95c4-4eff-b873-0beba327abea","Type":"ContainerDied","Data":"2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"}
Sep 07 16:31:40.051131 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.051077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:40.051310 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.051289    2577 scope.go:117] "RemoveContainer" containerID="2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"
Sep 07 16:31:40.052085 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.051936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:31:40.052340 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.052317    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:40.052396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.052380    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:40.052431 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.052410    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:31:40.522393 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.522363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:40.624656 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:40.624577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:41.060840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.060800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" event={"ID":"3d1fab42-95c4-4eff-b873-0beba327abea","Type":"ContainerStarted","Data":"dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4"}
Sep 07 16:31:41.086336 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.086306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:41.138938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.138912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:41.412011 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.411966    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:41.413645 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.413617    2577 patch_prober.go:28] interesting pod/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 07 16:31:41.413815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.413679    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 07 16:31:41.482975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.482934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:41.583298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.583264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:41.720656 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.720560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:41.831794 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:41.831482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:42.067540 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:42.064904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:42.412210 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:42.412163    2577 patch_prober.go:28] interesting pod/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 07 16:31:42.412446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:42.412249    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 07 16:31:42.952883 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:42.952857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:31:43.412092 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:43.412059    2577 patch_prober.go:28] interesting pod/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 07 16:31:43.412690 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:43.412659    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 07 16:31:44.411651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:44.411608    2577 patch_prober.go:28] interesting pod/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 07 16:31:44.411895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:44.411684    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 07 16:31:45.412274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:45.412223    2577 patch_prober.go:28] interesting pod/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 07 16:31:45.412780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:45.412302    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 07 16:31:45.907354 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:45.907314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:46.612479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:46.612452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:31:46.613869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:31:46.613844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:32:02.181901 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:02.181865    2577 generic.go:356] "Generic (PLEG): container finished" podID="985bac87-b1cf-47a5-964f-bf983c850c96" containerID="fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e" exitCode=0
Sep 07 16:32:02.182403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:02.181945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" event={"ID":"985bac87-b1cf-47a5-964f-bf983c850c96","Type":"ContainerDied","Data":"fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"}
Sep 07 16:32:02.182491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:02.182474    2577 scope.go:117] "RemoveContainer" containerID="fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"
Sep 07 16:32:02.183556 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:02.183528    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:32:02.183644 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:02.183599    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:32:02.183683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:02.183646    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:32:03.188287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:03.188250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" event={"ID":"985bac87-b1cf-47a5-964f-bf983c850c96","Type":"ContainerStarted","Data":"5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469"}
Sep 07 16:32:03.743957 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:03.743901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:32:03.744231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:03.744208    2577 patch_prober.go:28] interesting pod/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 07 16:32:03.744287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:03.744269    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 07 16:32:04.743971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:04.743935    2577 patch_prober.go:28] interesting pod/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 07 16:32:04.744353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:04.743994    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 07 16:32:05.743311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:05.743277    2577 patch_prober.go:28] interesting pod/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 07 16:32:05.743498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:05.743344    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 07 16:32:06.743810 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:06.743745    2577 patch_prober.go:28] interesting pod/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 07 16:32:06.744187 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:06.743834    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 07 16:32:07.225779 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.218575    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"]
Sep 07 16:32:07.241658 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.241628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"]
Sep 07 16:32:07.241993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.241975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.245776 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.245272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:07.245776 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.245747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-e4ed13ae-kserve-self-signed-certs\""
Sep 07 16:32:07.403427 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.403396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-dshm\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.403602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.403445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37caadee-8d36-4e38-ac20-1f0114a9a411-tls-certs\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.403602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.403515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-home\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.403602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.403535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-model-cache\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.403602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.403600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-tmp-dir\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.403738 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.403618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphx8\" (UniqueName: \"kubernetes.io/projected/37caadee-8d36-4e38-ac20-1f0114a9a411-kube-api-access-nphx8\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.504605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.504527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-dshm\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.504605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.504579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37caadee-8d36-4e38-ac20-1f0114a9a411-tls-certs\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.504869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.504617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-home\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.504869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.504631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-model-cache\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.504869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.504834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-tmp-dir\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.505030 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.504883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphx8\" (UniqueName: \"kubernetes.io/projected/37caadee-8d36-4e38-ac20-1f0114a9a411-kube-api-access-nphx8\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.505092 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.505041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-home\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.505375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.505246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-tmp-dir\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.505694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.505675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-model-cache\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.512774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.507474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-dshm\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.512774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.508106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37caadee-8d36-4e38-ac20-1f0114a9a411-tls-certs\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.516772 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.515067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphx8\" (UniqueName: \"kubernetes.io/projected/37caadee-8d36-4e38-ac20-1f0114a9a411-kube-api-access-nphx8\") pod \"e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") " pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.554477 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.554442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:07.743497 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.743461    2577 patch_prober.go:28] interesting pod/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 07 16:32:07.743670 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.743510    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 07 16:32:07.838817 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:07.838793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"]
Sep 07 16:32:07.841060 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:32:07.841022    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37caadee_8d36_4e38_ac20_1f0114a9a411.slice/crio-6098aacb65dd8ff88f0a4f5ff25c66763125284da5f90f6960d6d9089ad84788 WatchSource:0}: Error finding container 6098aacb65dd8ff88f0a4f5ff25c66763125284da5f90f6960d6d9089ad84788: Status 404 returned error can't find the container with id 6098aacb65dd8ff88f0a4f5ff25c66763125284da5f90f6960d6d9089ad84788
Sep 07 16:32:08.219782 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:08.219732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9" event={"ID":"37caadee-8d36-4e38-ac20-1f0114a9a411","Type":"ContainerStarted","Data":"9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150"}
Sep 07 16:32:08.219992 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:08.219796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9" event={"ID":"37caadee-8d36-4e38-ac20-1f0114a9a411","Type":"ContainerStarted","Data":"6098aacb65dd8ff88f0a4f5ff25c66763125284da5f90f6960d6d9089ad84788"}
Sep 07 16:32:08.238034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:08.237968    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9" podStartSLOduration=1.237942625 podStartE2EDuration="1.237942625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:08.235434636 +0000 UTC m=+733.116971606" watchObservedRunningTime="2026-09-07 16:32:08.237942625 +0000 UTC m=+733.119479604"
Sep 07 16:32:08.748111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:08.748071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:32:08.748541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:08.748520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:32:08.749234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:08.749216    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:32:11.938862 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:11.938823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:12.037984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.037949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:12.138780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.138716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:12.189658 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.189562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:12.288678 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.288647    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"]
Sep 07 16:32:12.288925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.288898    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy" containerID="cri-o://5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469" gracePeriod=30
Sep 07 16:32:12.301870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.301822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:12.697976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.697947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:13.000006 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:12.999877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:13.499656 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:13.499613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:15.986502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:15.986427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:16.077256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:16.077224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:17.184059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.184021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:17.286680 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.286646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:17.382238 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.382198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:17.539171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.539147    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:32:17.554556 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.554519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:17.554792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.554577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:17.572594 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.572564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:17.613012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.612986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-envoy\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-socket\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985bac87-b1cf-47a5-964f-bf983c850c96-istio-podinfo\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj85z\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-kube-api-access-gj85z\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613186    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985bac87-b1cf-47a5-964f-bf983c850c96-istiod-ca-cert\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-data\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:17.613456 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-credential-socket\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613508 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613471    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-data" (OuterVolumeSpecName: "istio-data") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:17.613508 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613502    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-certs\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613620 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-istio-token\") pod \"985bac87-b1cf-47a5-964f-bf983c850c96\" (UID: \"985bac87-b1cf-47a5-964f-bf983c850c96\") "
Sep 07 16:32:17.613675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613609    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:17.613675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985bac87-b1cf-47a5-964f-bf983c850c96-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:32:17.613675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613663    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:17.614022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.613985    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.614022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.614009    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.614022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.614023    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.614166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.614035    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/985bac87-b1cf-47a5-964f-bf983c850c96-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.614166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.614046    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.615474 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.615441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/985bac87-b1cf-47a5-964f-bf983c850c96-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:32:17.615723 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.615692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:17.615887 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.615789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-kube-api-access-gj85z" (OuterVolumeSpecName: "kube-api-access-gj85z") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "kube-api-access-gj85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:32:17.616004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.615984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-istio-token" (OuterVolumeSpecName: "istio-token") pod "985bac87-b1cf-47a5-964f-bf983c850c96" (UID: "985bac87-b1cf-47a5-964f-bf983c850c96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:32:17.708727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.708650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:17.714976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.714952    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/985bac87-b1cf-47a5-964f-bf983c850c96-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.714976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.714976    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj85z\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-kube-api-access-gj85z\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.715109 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.714992    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/985bac87-b1cf-47a5-964f-bf983c850c96-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.715109 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.715003    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/985bac87-b1cf-47a5-964f-bf983c850c96-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:17.814356 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.814319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:17.840735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.840692    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"]
Sep 07 16:32:17.841240 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.841224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy"
Sep 07 16:32:17.841286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.841243    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy"
Sep 07 16:32:17.841328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.841319    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy"
Sep 07 16:32:17.843629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.843613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:17.846282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.846262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b1b87a00-dockercfg-9t4wb\""
Sep 07 16:32:17.846410 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.846302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b1b87a00\""
Sep 07 16:32:17.856086 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.856064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"]
Sep 07 16:32:17.916349 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.916306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4ed737-a619-43bb-8fde-e31ac049c736-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") " pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:17.916569 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:17.916456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68bl\" (UniqueName: \"kubernetes.io/projected/cf4ed737-a619-43bb-8fde-e31ac049c736-kube-api-access-b68bl\") pod \"payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") " pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:18.017727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.017641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4ed737-a619-43bb-8fde-e31ac049c736-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") " pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:18.017727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.017699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68bl\" (UniqueName: \"kubernetes.io/projected/cf4ed737-a619-43bb-8fde-e31ac049c736-kube-api-access-b68bl\") pod \"payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") " pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:18.018382 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.018358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4ed737-a619-43bb-8fde-e31ac049c736-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") " pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:18.028804 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.028780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68bl\" (UniqueName: \"kubernetes.io/projected/cf4ed737-a619-43bb-8fde-e31ac049c736-kube-api-access-b68bl\") pod \"payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") " pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:18.155841 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.155808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:18.276892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.276789    2577 generic.go:356] "Generic (PLEG): container finished" podID="985bac87-b1cf-47a5-964f-bf983c850c96" containerID="5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469" exitCode=0
Sep 07 16:32:18.276892 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.276890    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"
Sep 07 16:32:18.277358 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.276917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" event={"ID":"985bac87-b1cf-47a5-964f-bf983c850c96","Type":"ContainerDied","Data":"5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469"}
Sep 07 16:32:18.277358 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.276971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn" event={"ID":"985bac87-b1cf-47a5-964f-bf983c850c96","Type":"ContainerDied","Data":"e199e7864564e9565b095dce6f63a0135480acc02ae687909d5fdad6a5a68fb6"}
Sep 07 16:32:18.277800 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.276994    2577 scope.go:117] "RemoveContainer" containerID="5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469"
Sep 07 16:32:18.287476 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.287458    2577 scope.go:117] "RemoveContainer" containerID="fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"
Sep 07 16:32:18.291589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.291568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:32:18.297133 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.297118    2577 scope.go:117] "RemoveContainer" containerID="5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469"
Sep 07 16:32:18.297493 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:32:18.297461    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469\": container with ID starting with 5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469 not found: ID does not exist" containerID="5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469"
Sep 07 16:32:18.297846 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.297515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469"} err="failed to get container status \"5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469\": rpc error: code = NotFound desc = could not find container \"5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469\": container with ID starting with 5c27737926f0dbe764289e00348f13d441f80e444397529ac0da3447dadcf469 not found: ID does not exist"
Sep 07 16:32:18.297846 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.297556    2577 scope.go:117] "RemoveContainer" containerID="fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"
Sep 07 16:32:18.298179 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:32:18.298091    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e\": container with ID starting with fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e not found: ID does not exist" containerID="fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"
Sep 07 16:32:18.298179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.298122    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e"} err="failed to get container status \"fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e\": rpc error: code = NotFound desc = could not find container \"fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e\": container with ID starting with fcad24345e06fa414118325a76e68776b5632df5891644d5004e5e10fc5b937e not found: ID does not exist"
Sep 07 16:32:18.301166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.301146    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"]
Sep 07 16:32:18.304274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.304253    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1e4cf2b6-gw-openshift-default-845f764bf6-v96cn"]
Sep 07 16:32:18.427851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:18.427829    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"]
Sep 07 16:32:18.430101 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:32:18.430076    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4ed737_a619_43bb_8fde_e31ac049c736.slice/crio-08be6ec9b5697f9dcc77d3b414449bc6ddd0953cb76512f9f0f3783a1e329d8f WatchSource:0}: Error finding container 08be6ec9b5697f9dcc77d3b414449bc6ddd0953cb76512f9f0f3783a1e329d8f: Status 404 returned error can't find the container with id 08be6ec9b5697f9dcc77d3b414449bc6ddd0953cb76512f9f0f3783a1e329d8f
Sep 07 16:32:19.284265 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:19.284231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l" event={"ID":"cf4ed737-a619-43bb-8fde-e31ac049c736","Type":"ContainerStarted","Data":"22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd"}
Sep 07 16:32:19.284687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:19.284289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l" event={"ID":"cf4ed737-a619-43bb-8fde-e31ac049c736","Type":"ContainerStarted","Data":"08be6ec9b5697f9dcc77d3b414449bc6ddd0953cb76512f9f0f3783a1e329d8f"}
Sep 07 16:32:19.284687 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:19.284348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:19.301050 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:19.301005    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l" podStartSLOduration=2.300989844 podStartE2EDuration="2.300989844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:19.298945773 +0000 UTC m=+744.180482746" watchObservedRunningTime="2026-09-07 16:32:19.300989844 +0000 UTC m=+744.182526812"
Sep 07 16:32:19.573709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:19.573609    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" path="/var/lib/kubelet/pods/985bac87-b1cf-47a5-964f-bf983c850c96/volumes"
Sep 07 16:32:20.015335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.015296    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"]
Sep 07 16:32:20.015853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.015837    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy"
Sep 07 16:32:20.015963 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.015855    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy"
Sep 07 16:32:20.016021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.015966    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="985bac87-b1cf-47a5-964f-bf983c850c96" containerName="istio-proxy"
Sep 07 16:32:20.019112 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.019094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.021631 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.021610    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8a3403\"/\"kube-root-ca.crt\""
Sep 07 16:32:20.021727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.021652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8a3403\"/\"openshift-service-ca.crt\""
Sep 07 16:32:20.023083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.023053    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8a3403\"/\"default-dockercfg-9s4pg\""
Sep 07 16:32:20.023198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.023166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8a3403\"/\"autores-model-8a3403-kserve-self-signed-certs\""
Sep 07 16:32:20.028273 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.028248    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"]
Sep 07 16:32:20.038895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.038862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-dshm\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.039026 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.038908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-model-cache\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.039026 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.038998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr55p\" (UniqueName: \"kubernetes.io/projected/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-kube-api-access-jr55p\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.039113 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.039079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-home\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.039205 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.039187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tmp-dir\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.039250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.039236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tls-certs\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-dshm\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-model-cache\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr55p\" (UniqueName: \"kubernetes.io/projected/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-kube-api-access-jr55p\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-home\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tmp-dir\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140512 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tls-certs\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-model-cache\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-home\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.140704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.140667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tmp-dir\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.142588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.142567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-dshm\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.142840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.142823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tls-certs\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.147586 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.147559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr55p\" (UniqueName: \"kubernetes.io/projected/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-kube-api-access-jr55p\") pod \"autores-model-8a3403-kserve-6d74db8559-jk564\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") " pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.320678 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.320579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:20.329572 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.329540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:20.602014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:20.601984    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"]
Sep 07 16:32:20.604031 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:32:20.604000    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cc431fb_a9e4_4fe0_9a4b_209693bfeb57.slice/crio-8627520963fcafeb31f2dbe8eb49aed3f6aeee0989f64bc87ffd1dde6f7f539b WatchSource:0}: Error finding container 8627520963fcafeb31f2dbe8eb49aed3f6aeee0989f64bc87ffd1dde6f7f539b: Status 404 returned error can't find the container with id 8627520963fcafeb31f2dbe8eb49aed3f6aeee0989f64bc87ffd1dde6f7f539b
Sep 07 16:32:21.300397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:21.300357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564" event={"ID":"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57","Type":"ContainerStarted","Data":"cb38da68c05dcf7e482f0f7b385386a448fdc8bd70340ea9fb945b7e36b834ac"}
Sep 07 16:32:21.300569 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:21.300409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564" event={"ID":"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57","Type":"ContainerStarted","Data":"8627520963fcafeb31f2dbe8eb49aed3f6aeee0989f64bc87ffd1dde6f7f539b"}
Sep 07 16:32:21.315869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:21.315817    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564" podStartSLOduration=2.315801217 podStartE2EDuration="2.315801217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:21.314569308 +0000 UTC m=+746.196106277" watchObservedRunningTime="2026-09-07 16:32:21.315801217 +0000 UTC m=+746.197338185"
Sep 07 16:32:30.294271 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:30.294235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:30.329994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:30.329962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:30.330147 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:30.330007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:30.342419 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:30.342391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:30.366379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:30.366350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:32:42.421179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.421146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"]
Sep 07 16:32:42.426357 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.426337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.429675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.429648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"]
Sep 07 16:32:42.430245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.430191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-747d6d\"/\"openshift-service-ca.crt\""
Sep 07 16:32:42.430245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.430221    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-747d6d\"/\"kube-root-ca.crt\""
Sep 07 16:32:42.430425 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.430221    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-747d6d\"/\"test-model-747d6d-kserve-self-signed-certs\""
Sep 07 16:32:42.430425 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.430226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-747d6d\"/\"default-dockercfg-w2rkt\""
Sep 07 16:32:42.443796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.443774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:42.572254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.572216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-tmp-dir\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.572436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.572270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-model-cache\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.572436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.572299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ca7402-f7c4-47f3-a025-0cdc03538402-tls-certs\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.572436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.572395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-home\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.572436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.572432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-dshm\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.572596 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.572459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf7b\" (UniqueName: \"kubernetes.io/projected/25ca7402-f7c4-47f3-a025-0cdc03538402-kube-api-access-gpf7b\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.673928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-tmp-dir\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.673974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-model-cache\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ca7402-f7c4-47f3-a025-0cdc03538402-tls-certs\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-home\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-dshm\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf7b\" (UniqueName: \"kubernetes.io/projected/25ca7402-f7c4-47f3-a025-0cdc03538402-kube-api-access-gpf7b\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-tmp-dir\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-home\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.674743 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.674715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-model-cache\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.676465 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.676442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-dshm\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.676583 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.676550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ca7402-f7c4-47f3-a025-0cdc03538402-tls-certs\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.682091 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.682066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf7b\" (UniqueName: \"kubernetes.io/projected/25ca7402-f7c4-47f3-a025-0cdc03538402-kube-api-access-gpf7b\") pod \"test-model-747d6d-kserve-fd78bf465-jvk9r\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") " pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:42.737860 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:42.737825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:43.007433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:43.007408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"]
Sep 07 16:32:43.009414 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:32:43.009392    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25ca7402_f7c4_47f3_a025_0cdc03538402.slice/crio-5c209e5658c985c1916ed709112ef820b29a2ed41a39f2e0065f05891148781c WatchSource:0}: Error finding container 5c209e5658c985c1916ed709112ef820b29a2ed41a39f2e0065f05891148781c: Status 404 returned error can't find the container with id 5c209e5658c985c1916ed709112ef820b29a2ed41a39f2e0065f05891148781c
Sep 07 16:32:43.427644 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:43.427607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" event={"ID":"25ca7402-f7c4-47f3-a025-0cdc03538402","Type":"ContainerStarted","Data":"442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a"}
Sep 07 16:32:43.428045 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:43.427651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" event={"ID":"25ca7402-f7c4-47f3-a025-0cdc03538402","Type":"ContainerStarted","Data":"5c209e5658c985c1916ed709112ef820b29a2ed41a39f2e0065f05891148781c"}
Sep 07 16:32:43.443669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:43.443622    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" podStartSLOduration=1.44360613 podStartE2EDuration="1.44360613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:43.442341171 +0000 UTC m=+768.323878141" watchObservedRunningTime="2026-09-07 16:32:43.44360613 +0000 UTC m=+768.325143100"
Sep 07 16:32:44.618616 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:44.618571    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"]
Sep 07 16:32:44.636465 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:44.636427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:45.013999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:45.013949    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"]
Sep 07 16:32:45.036640 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:45.036606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:46.262921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:46.262890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"]
Sep 07 16:32:46.279890 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:46.279848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:46.485081 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:46.485027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:47.563460 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.563422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:47.587891 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.587860    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"]
Sep 07 16:32:47.588146 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.588111    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l" podUID="cf4ed737-a619-43bb-8fde-e31ac049c736" containerName="payload-processing" containerID="cri-o://22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd" gracePeriod=30
Sep 07 16:32:47.663542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.663511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:47.752242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.752212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:47.821720 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.821687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:47.844955 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:47.844930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:48.032074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.032041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4ed737-a619-43bb-8fde-e31ac049c736-plugins-config-volume\") pod \"cf4ed737-a619-43bb-8fde-e31ac049c736\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") "
Sep 07 16:32:48.032301 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.032218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b68bl\" (UniqueName: \"kubernetes.io/projected/cf4ed737-a619-43bb-8fde-e31ac049c736-kube-api-access-b68bl\") pod \"cf4ed737-a619-43bb-8fde-e31ac049c736\" (UID: \"cf4ed737-a619-43bb-8fde-e31ac049c736\") "
Sep 07 16:32:48.032518 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.032490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4ed737-a619-43bb-8fde-e31ac049c736-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf4ed737-a619-43bb-8fde-e31ac049c736" (UID: "cf4ed737-a619-43bb-8fde-e31ac049c736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:32:48.032626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.032611    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4ed737-a619-43bb-8fde-e31ac049c736-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:48.034551 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.034525    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4ed737-a619-43bb-8fde-e31ac049c736-kube-api-access-b68bl" (OuterVolumeSpecName: "kube-api-access-b68bl") pod "cf4ed737-a619-43bb-8fde-e31ac049c736" (UID: "cf4ed737-a619-43bb-8fde-e31ac049c736"). InnerVolumeSpecName "kube-api-access-b68bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:32:48.133872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.133743    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68bl\" (UniqueName: \"kubernetes.io/projected/cf4ed737-a619-43bb-8fde-e31ac049c736-kube-api-access-b68bl\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:48.454207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.454175    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf4ed737-a619-43bb-8fde-e31ac049c736" containerID="22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd" exitCode=0
Sep 07 16:32:48.454412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.454221    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"
Sep 07 16:32:48.454412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.454258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l" event={"ID":"cf4ed737-a619-43bb-8fde-e31ac049c736","Type":"ContainerDied","Data":"22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd"}
Sep 07 16:32:48.454412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.454314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l" event={"ID":"cf4ed737-a619-43bb-8fde-e31ac049c736","Type":"ContainerDied","Data":"08be6ec9b5697f9dcc77d3b414449bc6ddd0953cb76512f9f0f3783a1e329d8f"}
Sep 07 16:32:48.454412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.454336    2577 scope.go:117] "RemoveContainer" containerID="22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd"
Sep 07 16:32:48.463940 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.463925    2577 scope.go:117] "RemoveContainer" containerID="22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd"
Sep 07 16:32:48.464217 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:32:48.464201    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd\": container with ID starting with 22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd not found: ID does not exist" containerID="22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd"
Sep 07 16:32:48.464260 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.464227    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd"} err="failed to get container status \"22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd\": rpc error: code = NotFound desc = could not find container \"22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd\": container with ID starting with 22e48869a91d7039d773c957849b947210ed48cea5f0daba22973ac0a7aaa4cd not found: ID does not exist"
Sep 07 16:32:48.475929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.475906    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"]
Sep 07 16:32:48.477944 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.477920    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1b87a00-776dbbb84f-68d7l"]
Sep 07 16:32:48.837531 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:48.837430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:49.236889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:49.236862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:49.516204 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:49.516083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:49.569647 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:49.569617    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf4ed737-a619-43bb-8fde-e31ac049c736" path="/var/lib/kubelet/pods/cf4ed737-a619-43bb-8fde-e31ac049c736/volumes"
Sep 07 16:32:49.615626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:49.615591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:52.271141 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:52.271106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:52.361305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:52.361270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:52.738926 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:52.738888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:52.738926 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:52.738934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:52.753435 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:52.753409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:53.498075 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:53.498047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:32:53.507999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:53.507936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:53.610522 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:53.610494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:53.708332 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:53.708301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:54.209430 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:54.209385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:54.309519 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:54.309489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:56.523616 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:56.523569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:57.060782 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.060737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:57.087783 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.087728    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"]
Sep 07 16:32:57.088032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.087988    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm" podUID="a2d85d3c-b7a2-40c5-99a0-9b2166bac792" containerName="payload-pre-processing" containerID="cri-o://e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d" gracePeriod=30
Sep 07 16:32:57.161277 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.161238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:57.251538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.251502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:57.307389 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.307358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:57.341742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.341717    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:32:57.433898 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.433862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-626g7\" (UniqueName: \"kubernetes.io/projected/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-kube-api-access-626g7\") pod \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") "
Sep 07 16:32:57.434070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.434017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-plugins-config-volume\") pod \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\" (UID: \"a2d85d3c-b7a2-40c5-99a0-9b2166bac792\") "
Sep 07 16:32:57.434383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.434354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2d85d3c-b7a2-40c5-99a0-9b2166bac792" (UID: "a2d85d3c-b7a2-40c5-99a0-9b2166bac792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:32:57.436249 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.436225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-kube-api-access-626g7" (OuterVolumeSpecName: "kube-api-access-626g7") pod "a2d85d3c-b7a2-40c5-99a0-9b2166bac792" (UID: "a2d85d3c-b7a2-40c5-99a0-9b2166bac792"). InnerVolumeSpecName "kube-api-access-626g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:32:57.505294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.505257    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2d85d3c-b7a2-40c5-99a0-9b2166bac792" containerID="e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d" exitCode=0
Sep 07 16:32:57.505470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.505304    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"
Sep 07 16:32:57.505470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.505339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm" event={"ID":"a2d85d3c-b7a2-40c5-99a0-9b2166bac792","Type":"ContainerDied","Data":"e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d"}
Sep 07 16:32:57.505470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.505403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm" event={"ID":"a2d85d3c-b7a2-40c5-99a0-9b2166bac792","Type":"ContainerDied","Data":"d001d0600b04f79ce504320d615dca4daa4ffbd6dc8c4f14ece1e5eb465ba4f4"}
Sep 07 16:32:57.505470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.505426    2577 scope.go:117] "RemoveContainer" containerID="e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d"
Sep 07 16:32:57.514894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.514871    2577 scope.go:117] "RemoveContainer" containerID="e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d"
Sep 07 16:32:57.515191 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:32:57.515174    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d\": container with ID starting with e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d not found: ID does not exist" containerID="e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d"
Sep 07 16:32:57.515249 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.515200    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d"} err="failed to get container status \"e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d\": rpc error: code = NotFound desc = could not find container \"e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d\": container with ID starting with e864f4b7980ddb47624c46b21d41b6e0d74b2b1849efa80251d2f6767f2d801d not found: ID does not exist"
Sep 07 16:32:57.527729 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.527699    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"]
Sep 07 16:32:57.531638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.531614    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8a3403-674cdc45f4-fhqlm"]
Sep 07 16:32:57.534997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.534976    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:57.535058 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.534999    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626g7\" (UniqueName: \"kubernetes.io/projected/a2d85d3c-b7a2-40c5-99a0-9b2166bac792-kube-api-access-626g7\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:57.568493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.568412    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d85d3c-b7a2-40c5-99a0-9b2166bac792" path="/var/lib/kubelet/pods/a2d85d3c-b7a2-40c5-99a0-9b2166bac792/volumes"
Sep 07 16:32:57.579854 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:57.579825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:32:59.641171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.641135    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps" podUID="ef34115e-4a33-426a-8fed-63984218bab2" containerName="main" containerID="cri-o://72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50" gracePeriod=45
Sep 07 16:32:59.891108 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.891084    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:32:59.960061 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.959975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gtcq\" (UniqueName: \"kubernetes.io/projected/ef34115e-4a33-426a-8fed-63984218bab2-kube-api-access-7gtcq\") pod \"ef34115e-4a33-426a-8fed-63984218bab2\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") "
Sep 07 16:32:59.960061 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-dshm\") pod \"ef34115e-4a33-426a-8fed-63984218bab2\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") "
Sep 07 16:32:59.960061 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960064    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-home\") pod \"ef34115e-4a33-426a-8fed-63984218bab2\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") "
Sep 07 16:32:59.960309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-model-cache\") pod \"ef34115e-4a33-426a-8fed-63984218bab2\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") "
Sep 07 16:32:59.960309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-tmp-dir\") pod \"ef34115e-4a33-426a-8fed-63984218bab2\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") "
Sep 07 16:32:59.960309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34115e-4a33-426a-8fed-63984218bab2-tls-certs\") pod \"ef34115e-4a33-426a-8fed-63984218bab2\" (UID: \"ef34115e-4a33-426a-8fed-63984218bab2\") "
Sep 07 16:32:59.960462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-home" (OuterVolumeSpecName: "home") pod "ef34115e-4a33-426a-8fed-63984218bab2" (UID: "ef34115e-4a33-426a-8fed-63984218bab2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:59.960462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960417    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef34115e-4a33-426a-8fed-63984218bab2" (UID: "ef34115e-4a33-426a-8fed-63984218bab2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:59.960462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef34115e-4a33-426a-8fed-63984218bab2" (UID: "ef34115e-4a33-426a-8fed-63984218bab2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:59.960651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960629    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:59.960697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960658    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:59.960697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.960671    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:32:59.962525 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.962499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-dshm" (OuterVolumeSpecName: "dshm") pod "ef34115e-4a33-426a-8fed-63984218bab2" (UID: "ef34115e-4a33-426a-8fed-63984218bab2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:32:59.962625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.962586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef34115e-4a33-426a-8fed-63984218bab2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef34115e-4a33-426a-8fed-63984218bab2" (UID: "ef34115e-4a33-426a-8fed-63984218bab2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:32:59.962625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:32:59.962603    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef34115e-4a33-426a-8fed-63984218bab2-kube-api-access-7gtcq" (OuterVolumeSpecName: "kube-api-access-7gtcq") pod "ef34115e-4a33-426a-8fed-63984218bab2" (UID: "ef34115e-4a33-426a-8fed-63984218bab2"). InnerVolumeSpecName "kube-api-access-7gtcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:33:00.036932 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.036886    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9" podUID="37caadee-8d36-4e38-ac20-1f0114a9a411" containerName="main" containerID="cri-o://9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150" gracePeriod=45
Sep 07 16:33:00.062012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.061973    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtcq\" (UniqueName: \"kubernetes.io/projected/ef34115e-4a33-426a-8fed-63984218bab2-kube-api-access-7gtcq\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.062012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.062007    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34115e-4a33-426a-8fed-63984218bab2-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.062012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.062017    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34115e-4a33-426a-8fed-63984218bab2-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.281856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.281827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:33:00.364608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-home\") pod \"37caadee-8d36-4e38-ac20-1f0114a9a411\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") "
Sep 07 16:33:00.364608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37caadee-8d36-4e38-ac20-1f0114a9a411-tls-certs\") pod \"37caadee-8d36-4e38-ac20-1f0114a9a411\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") "
Sep 07 16:33:00.364913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-model-cache\") pod \"37caadee-8d36-4e38-ac20-1f0114a9a411\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") "
Sep 07 16:33:00.364913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-tmp-dir\") pod \"37caadee-8d36-4e38-ac20-1f0114a9a411\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") "
Sep 07 16:33:00.364913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-dshm\") pod \"37caadee-8d36-4e38-ac20-1f0114a9a411\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") "
Sep 07 16:33:00.364913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nphx8\" (UniqueName: \"kubernetes.io/projected/37caadee-8d36-4e38-ac20-1f0114a9a411-kube-api-access-nphx8\") pod \"37caadee-8d36-4e38-ac20-1f0114a9a411\" (UID: \"37caadee-8d36-4e38-ac20-1f0114a9a411\") "
Sep 07 16:33:00.365099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-model-cache" (OuterVolumeSpecName: "model-cache") pod "37caadee-8d36-4e38-ac20-1f0114a9a411" (UID: "37caadee-8d36-4e38-ac20-1f0114a9a411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:00.365099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.364957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37caadee-8d36-4e38-ac20-1f0114a9a411" (UID: "37caadee-8d36-4e38-ac20-1f0114a9a411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:00.365182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.365150    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.365182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.365167    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.365303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.365189    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-home" (OuterVolumeSpecName: "home") pod "37caadee-8d36-4e38-ac20-1f0114a9a411" (UID: "37caadee-8d36-4e38-ac20-1f0114a9a411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:00.367003 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.366979    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-dshm" (OuterVolumeSpecName: "dshm") pod "37caadee-8d36-4e38-ac20-1f0114a9a411" (UID: "37caadee-8d36-4e38-ac20-1f0114a9a411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:00.367107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.367030    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37caadee-8d36-4e38-ac20-1f0114a9a411-kube-api-access-nphx8" (OuterVolumeSpecName: "kube-api-access-nphx8") pod "37caadee-8d36-4e38-ac20-1f0114a9a411" (UID: "37caadee-8d36-4e38-ac20-1f0114a9a411"). InnerVolumeSpecName "kube-api-access-nphx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:33:00.367107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.367051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37caadee-8d36-4e38-ac20-1f0114a9a411-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37caadee-8d36-4e38-ac20-1f0114a9a411" (UID: "37caadee-8d36-4e38-ac20-1f0114a9a411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:33:00.466294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.466253    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.466294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.466285    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphx8\" (UniqueName: \"kubernetes.io/projected/37caadee-8d36-4e38-ac20-1f0114a9a411-kube-api-access-nphx8\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.466294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.466296    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37caadee-8d36-4e38-ac20-1f0114a9a411-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.466294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.466304    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37caadee-8d36-4e38-ac20-1f0114a9a411-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:00.523631 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.523599    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef34115e-4a33-426a-8fed-63984218bab2" containerID="72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50" exitCode=0
Sep 07 16:33:00.523814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.523645    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"
Sep 07 16:33:00.523814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.523681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps" event={"ID":"ef34115e-4a33-426a-8fed-63984218bab2","Type":"ContainerDied","Data":"72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50"}
Sep 07 16:33:00.523814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.523718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps" event={"ID":"ef34115e-4a33-426a-8fed-63984218bab2","Type":"ContainerDied","Data":"390ea7eec98d8af8f9c74db4bf9fb3715624dd35a0ed64d5d40051ac77d8e484"}
Sep 07 16:33:00.523814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.523744    2577 scope.go:117] "RemoveContainer" containerID="72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50"
Sep 07 16:33:00.525250 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.525222    2577 generic.go:356] "Generic (PLEG): container finished" podID="37caadee-8d36-4e38-ac20-1f0114a9a411" containerID="9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150" exitCode=0
Sep 07 16:33:00.525366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.525268    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"
Sep 07 16:33:00.525366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.525298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9" event={"ID":"37caadee-8d36-4e38-ac20-1f0114a9a411","Type":"ContainerDied","Data":"9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150"}
Sep 07 16:33:00.525366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.525334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9" event={"ID":"37caadee-8d36-4e38-ac20-1f0114a9a411","Type":"ContainerDied","Data":"6098aacb65dd8ff88f0a4f5ff25c66763125284da5f90f6960d6d9089ad84788"}
Sep 07 16:33:00.535875 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.535847    2577 scope.go:117] "RemoveContainer" containerID="72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50"
Sep 07 16:33:00.536511 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:33:00.536481    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50\": container with ID starting with 72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50 not found: ID does not exist" containerID="72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50"
Sep 07 16:33:00.536631 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.536528    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50"} err="failed to get container status \"72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50\": rpc error: code = NotFound desc = could not find container \"72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50\": container with ID starting with 72155604f9da998c87ae67400b93ae18cd1d3bee3fa98c532771157368bbbe50 not found: ID does not exist"
Sep 07 16:33:00.536631 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.536553    2577 scope.go:117] "RemoveContainer" containerID="9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150"
Sep 07 16:33:00.549517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.549485    2577 scope.go:117] "RemoveContainer" containerID="9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150"
Sep 07 16:33:00.551010 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:33:00.550976    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150\": container with ID starting with 9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150 not found: ID does not exist" containerID="9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150"
Sep 07 16:33:00.551129 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.551018    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150"} err="failed to get container status \"9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150\": rpc error: code = NotFound desc = could not find container \"9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150\": container with ID starting with 9bbe0662fdbb1d456b6751b4ced88d1642c52d3c055aef32446febc5830ef150 not found: ID does not exist"
Sep 07 16:33:00.552163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.552141    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"]
Sep 07 16:33:00.554232 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.554210    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-e4ed13ae-kserve-5b8c665b7-c45ps"]
Sep 07 16:33:00.563458 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.563311    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"]
Sep 07 16:33:00.565574 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:00.565545    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-e4ed13ae-kserve-77bb7d76c7-ffhv9"]
Sep 07 16:33:01.284287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.284240    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564" podUID="0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" containerName="main" containerID="cri-o://cb38da68c05dcf7e482f0f7b385386a448fdc8bd70340ea9fb945b7e36b834ac" gracePeriod=45
Sep 07 16:33:01.536093 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.536013    2577 generic.go:356] "Generic (PLEG): container finished" podID="0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" containerID="cb38da68c05dcf7e482f0f7b385386a448fdc8bd70340ea9fb945b7e36b834ac" exitCode=0
Sep 07 16:33:01.536231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.536082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564" event={"ID":"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57","Type":"ContainerDied","Data":"cb38da68c05dcf7e482f0f7b385386a448fdc8bd70340ea9fb945b7e36b834ac"}
Sep 07 16:33:01.557955 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.557916    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:33:01.569344 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.569310    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37caadee-8d36-4e38-ac20-1f0114a9a411" path="/var/lib/kubelet/pods/37caadee-8d36-4e38-ac20-1f0114a9a411/volumes"
Sep 07 16:33:01.569780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.569744    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef34115e-4a33-426a-8fed-63984218bab2" path="/var/lib/kubelet/pods/ef34115e-4a33-426a-8fed-63984218bab2/volumes"
Sep 07 16:33:01.680842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.680800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr55p\" (UniqueName: \"kubernetes.io/projected/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-kube-api-access-jr55p\") pod \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") "
Sep 07 16:33:01.681041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.680858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-dshm\") pod \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") "
Sep 07 16:33:01.681041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.680961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-home\") pod \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") "
Sep 07 16:33:01.681041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-model-cache\") pod \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") "
Sep 07 16:33:01.681206 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tmp-dir\") pod \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") "
Sep 07 16:33:01.681206 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681145    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tls-certs\") pod \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\" (UID: \"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57\") "
Sep 07 16:33:01.681318 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-home" (OuterVolumeSpecName: "home") pod "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" (UID: "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:01.681318 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681272    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" (UID: "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:01.681475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" (UID: "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:01.681659 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681615    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:01.681659 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681652    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:01.681875 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.681731    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:01.683943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.683913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-dshm" (OuterVolumeSpecName: "dshm") pod "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" (UID: "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:33:01.683943 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.683932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" (UID: "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:33:01.684088 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.684032    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-kube-api-access-jr55p" (OuterVolumeSpecName: "kube-api-access-jr55p") pod "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" (UID: "0cc431fb-a9e4-4fe0-9a4b-209693bfeb57"). InnerVolumeSpecName "kube-api-access-jr55p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:33:01.783043 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.782985    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:01.783043 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.783033    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr55p\" (UniqueName: \"kubernetes.io/projected/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-kube-api-access-jr55p\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:01.783043 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:01.783044    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:33:02.542491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:02.542456    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"
Sep 07 16:33:02.542975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:02.542490    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564" event={"ID":"0cc431fb-a9e4-4fe0-9a4b-209693bfeb57","Type":"ContainerDied","Data":"8627520963fcafeb31f2dbe8eb49aed3f6aeee0989f64bc87ffd1dde6f7f539b"}
Sep 07 16:33:02.542975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:02.542543    2577 scope.go:117] "RemoveContainer" containerID="cb38da68c05dcf7e482f0f7b385386a448fdc8bd70340ea9fb945b7e36b834ac"
Sep 07 16:33:02.564083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:02.564049    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"]
Sep 07 16:33:02.572825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:02.572797    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-8a3403/autores-model-8a3403-kserve-6d74db8559-jk564"]
Sep 07 16:33:02.656965 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:02.656924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:03.501957 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:03.501927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:03.568626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:03.568592    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" path="/var/lib/kubelet/pods/0cc431fb-a9e4-4fe0-9a4b-209693bfeb57/volumes"
Sep 07 16:33:03.602293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:03.602265    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:03.742635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:03.742600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:04.125505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.125466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:04.394370 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394340    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"]
Sep 07 16:33:04.394915 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394880    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef34115e-4a33-426a-8fed-63984218bab2" containerName="main"
Sep 07 16:33:04.394915 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394903    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef34115e-4a33-426a-8fed-63984218bab2" containerName="main"
Sep 07 16:33:04.394915 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394918    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37caadee-8d36-4e38-ac20-1f0114a9a411" containerName="main"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394925    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37caadee-8d36-4e38-ac20-1f0114a9a411" containerName="main"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394948    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d85d3c-b7a2-40c5-99a0-9b2166bac792" containerName="payload-pre-processing"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394958    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d85d3c-b7a2-40c5-99a0-9b2166bac792" containerName="payload-pre-processing"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394976    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" containerName="main"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.394984    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" containerName="main"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395005    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4ed737-a619-43bb-8fde-e31ac049c736" containerName="payload-processing"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395013    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4ed737-a619-43bb-8fde-e31ac049c736" containerName="payload-processing"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395098    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc431fb-a9e4-4fe0-9a4b-209693bfeb57" containerName="main"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395109    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d85d3c-b7a2-40c5-99a0-9b2166bac792" containerName="payload-pre-processing"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395116    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37caadee-8d36-4e38-ac20-1f0114a9a411" containerName="main"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395124    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4ed737-a619-43bb-8fde-e31ac049c736" containerName="payload-processing"
Sep 07 16:33:04.395172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.395133    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef34115e-4a33-426a-8fed-63984218bab2" containerName="main"
Sep 07 16:33:04.400246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.400208    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.402727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.402704    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-cc0983-openshift-default-dockercfg-ttf2x\""
Sep 07 16:33:04.412747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.412721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"]
Sep 07 16:33:04.435396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.435362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:04.511354 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istiod-ca-cert\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.511707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cl77\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-kube-api-access-9cl77\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.511707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-token\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.511707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-data\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.511938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-certs\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.511938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-podinfo\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.511938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-credential-socket\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.512094 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.511973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-socket\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.512094 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.512017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-envoy\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.569446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.569414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:04.612835 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.612802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istiod-ca-cert\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.612849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cl77\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-kube-api-access-9cl77\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.612878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-token\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.612899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-data\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.612918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-certs\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.612937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-podinfo\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-credential-socket\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-socket\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613287 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-envoy\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-certs\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-data\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613592 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-socket\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istiod-ca-cert\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.613706 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.613692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-credential-socket\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.615564 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.615540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-podinfo\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.615657 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.615593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-envoy\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.620377 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.620355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-token\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.620698 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.620680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cl77\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-kube-api-access-9cl77\") pod \"e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") " pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.712494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.712392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:04.990030 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.989814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"]
Sep 07 16:33:04.993169 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:33:04.993139    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536bd826_5cfd_4a62_8b3b_a9729f4bc74a.slice/crio-162e7d4b21b359b0f11deecf993a0b1bde6c748c0297fffc252043620f731340 WatchSource:0}: Error finding container 162e7d4b21b359b0f11deecf993a0b1bde6c748c0297fffc252043620f731340: Status 404 returned error can't find the container with id 162e7d4b21b359b0f11deecf993a0b1bde6c748c0297fffc252043620f731340
Sep 07 16:33:04.995722 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.995691    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:33:04.995827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.995777    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:33:04.995827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:04.995808    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:33:05.572436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:05.572402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" event={"ID":"536bd826-5cfd-4a62-8b3b-a9729f4bc74a","Type":"ContainerStarted","Data":"1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"}
Sep 07 16:33:05.572843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:05.572462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" event={"ID":"536bd826-5cfd-4a62-8b3b-a9729f4bc74a","Type":"ContainerStarted","Data":"162e7d4b21b359b0f11deecf993a0b1bde6c748c0297fffc252043620f731340"}
Sep 07 16:33:05.591017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:05.590965    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podStartSLOduration=1.590947902 podStartE2EDuration="1.590947902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:33:05.589739394 +0000 UTC m=+790.471276365" watchObservedRunningTime="2026-09-07 16:33:05.590947902 +0000 UTC m=+790.472484871"
Sep 07 16:33:05.713100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:05.713062    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:06.718392 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:06.718358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:07.485689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:07.485650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:07.578585 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:07.578556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:07.579666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:07.579639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:07.590173 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:07.590145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:08.647223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:08.647194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:08.747771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:08.747725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:08.867146 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:08.867116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:09.223099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.223057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:09.314620 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.314589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:09.391711 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.391676    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"]
Sep 07 16:33:09.395493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.395475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.398110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.398088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cc0983-dockercfg-b4bls\""
Sep 07 16:33:09.398110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.398107    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cc0983\""
Sep 07 16:33:09.405905 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.405883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"]
Sep 07 16:33:09.565563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.565469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjk2\" (UniqueName: \"kubernetes.io/projected/4f12face-2a96-4874-97dc-dde4c22b3017-kube-api-access-gfjk2\") pod \"payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.565563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.565524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f12face-2a96-4874-97dc-dde4c22b3017-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.666840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.666799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjk2\" (UniqueName: \"kubernetes.io/projected/4f12face-2a96-4874-97dc-dde4c22b3017-kube-api-access-gfjk2\") pod \"payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.667262 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.666861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f12face-2a96-4874-97dc-dde4c22b3017-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.667498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.667479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f12face-2a96-4874-97dc-dde4c22b3017-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.674520 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.674487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjk2\" (UniqueName: \"kubernetes.io/projected/4f12face-2a96-4874-97dc-dde4c22b3017-kube-api-access-gfjk2\") pod \"payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.707522 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.707487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:09.980583 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:09.980555    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"]
Sep 07 16:33:09.982795 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:33:09.982742    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f12face_2a96_4874_97dc_dde4c22b3017.slice/crio-ab3f11d1df3a9084f0b9454ae9840337f9211e20de017c38168d3dbddeddc123 WatchSource:0}: Error finding container ab3f11d1df3a9084f0b9454ae9840337f9211e20de017c38168d3dbddeddc123: Status 404 returned error can't find the container with id ab3f11d1df3a9084f0b9454ae9840337f9211e20de017c38168d3dbddeddc123
Sep 07 16:33:10.601529 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:10.601491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb" event={"ID":"4f12face-2a96-4874-97dc-dde4c22b3017","Type":"ContainerStarted","Data":"3177974823dfaf7e24e3318b9ba7f793956808d562411b2c082fc35956df6a52"}
Sep 07 16:33:10.601713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:10.601564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb" event={"ID":"4f12face-2a96-4874-97dc-dde4c22b3017","Type":"ContainerStarted","Data":"ab3f11d1df3a9084f0b9454ae9840337f9211e20de017c38168d3dbddeddc123"}
Sep 07 16:33:10.601713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:10.601606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:10.617547 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:10.617491    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb" podStartSLOduration=1.617475664 podStartE2EDuration="1.617475664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:33:10.616270369 +0000 UTC m=+795.497807319" watchObservedRunningTime="2026-09-07 16:33:10.617475664 +0000 UTC m=+795.499012633"
Sep 07 16:33:14.916605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:14.916566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:15.558730 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:15.558696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:16.202072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:16.202038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:21.611015 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:21.610924    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:33:23.877072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:23.877042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:23.949232 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:23.949194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:24.028511 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:24.028456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:24.084233 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:24.084201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:24.200615 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:24.200582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:24.837696 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:24.837667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:24.932849 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:24.932815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:25.037188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:25.037158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:26.461775 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:26.461727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:29.706589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:29.706554    2577 generic.go:356] "Generic (PLEG): container finished" podID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerID="1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991" exitCode=0
Sep 07 16:33:29.707048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:29.706634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" event={"ID":"536bd826-5cfd-4a62-8b3b-a9729f4bc74a","Type":"ContainerDied","Data":"1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"}
Sep 07 16:33:29.707125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:29.707108    2577 scope.go:117] "RemoveContainer" containerID="1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"
Sep 07 16:33:29.708380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:29.708348    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:33:29.708467 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:29.708421    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:33:29.708467 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:29.708457    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:33:30.075070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:30.074980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:30.719119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:30.719077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" event={"ID":"536bd826-5cfd-4a62-8b3b-a9729f4bc74a","Type":"ContainerStarted","Data":"ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658"}
Sep 07 16:33:31.713248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:31.713191    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:31.713542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:31.713513    2577 patch_prober.go:28] interesting pod/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 07 16:33:31.713638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:31.713574    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 07 16:33:32.135054 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.134964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:32.196702 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.196662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:32.266500 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.266472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:32.367554 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.367503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:32.466458 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.466427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:32.713780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.713093    2577 patch_prober.go:28] interesting pod/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 07 16:33:32.713780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.713149    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 07 16:33:32.773026 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:32.772944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:33.713981 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:33.713477    2577 patch_prober.go:28] interesting pod/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 07 16:33:33.713981 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:33.713545    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 07 16:33:34.262991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:34.262954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:34.713889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:34.713851    2577 patch_prober.go:28] interesting pod/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 07 16:33:34.714193 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:34.714166    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 07 16:33:35.713155 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:35.713117    2577 patch_prober.go:28] interesting pod/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 07 16:33:35.713484 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:35.713438    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 07 16:33:36.717699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:36.717668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:36.718119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:36.718013    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:36.718856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:36.718834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:33:46.530656 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:46.530620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:46.630869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:46.630837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:51.330871 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.330837    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"]
Sep 07 16:33:51.334528 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.334505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.337038 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.337017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cc0983\"/\"default-dockercfg-258bg\""
Sep 07 16:33:51.338837 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.338443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cc0983\"/\"explicit-model-cc0983-kserve-self-signed-certs\""
Sep 07 16:33:51.338837 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.338664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cc0983\"/\"openshift-service-ca.crt\""
Sep 07 16:33:51.338837 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.338796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cc0983\"/\"kube-root-ca.crt\""
Sep 07 16:33:51.341306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.341282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"]
Sep 07 16:33:51.353398 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.353345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:33:51.401034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.400960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245x4\" (UniqueName: \"kubernetes.io/projected/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-kube-api-access-245x4\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.401224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.401054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-model-cache\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.401224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.401078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tls-certs\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.401224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.401168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-dshm\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.401224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.401206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-home\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.401456 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.401308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tmp-dir\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.501903 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.501867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tmp-dir\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.501937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245x4\" (UniqueName: \"kubernetes.io/projected/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-kube-api-access-245x4\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.501964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-model-cache\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.501991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tls-certs\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.502040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-dshm\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.502069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-home\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.502318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tmp-dir\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.502335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-model-cache\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.502485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.502445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-home\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.504468 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.504447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-dshm\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.504591 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.504572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tls-certs\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.510632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.510604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-245x4\" (UniqueName: \"kubernetes.io/projected/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-kube-api-access-245x4\") pod \"explicit-model-cc0983-kserve-ccc894657-dkgx9\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") " pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.645819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.645787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:33:51.919258 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:51.919230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"]
Sep 07 16:33:51.920779 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:33:51.920732    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf431e2cd_a2ed_4aef_b5f4_05acc82aaaf4.slice/crio-67261a3ef82d105a4b2c63eaf6cd57b003d1ffe1007de0a9ac16c766ab6a1349 WatchSource:0}: Error finding container 67261a3ef82d105a4b2c63eaf6cd57b003d1ffe1007de0a9ac16c766ab6a1349: Status 404 returned error can't find the container with id 67261a3ef82d105a4b2c63eaf6cd57b003d1ffe1007de0a9ac16c766ab6a1349
Sep 07 16:33:52.841634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:52.841602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9" event={"ID":"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4","Type":"ContainerStarted","Data":"f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686"}
Sep 07 16:33:52.841634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:52.841641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9" event={"ID":"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4","Type":"ContainerStarted","Data":"67261a3ef82d105a4b2c63eaf6cd57b003d1ffe1007de0a9ac16c766ab6a1349"}
Sep 07 16:33:52.858968 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:52.858914    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9" podStartSLOduration=1.858894742 podStartE2EDuration="1.858894742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:33:52.857459336 +0000 UTC m=+837.738996307" watchObservedRunningTime="2026-09-07 16:33:52.858894742 +0000 UTC m=+837.740431708"
Sep 07 16:33:58.688502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:33:58.688464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:01.646259 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:01.646218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:34:01.646820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:01.646288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:34:01.659481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:01.659453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:34:01.903298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:01.903205    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:34:04.322370 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:04.322335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:04.390330 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:04.390293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:04.471146 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:04.471113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:17.160472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:17.160437    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"]
Sep 07 16:34:17.185683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:17.185644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:17.371168 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:17.371126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:20.285826 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:20.285796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:25.955428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:25.955389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:26.062303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:26.062263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:26.156070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:26.156043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:27.869177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:27.869143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:27.913683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:27.913648    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"]
Sep 07 16:34:27.913956 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:27.913929    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb" podUID="4f12face-2a96-4874-97dc-dde4c22b3017" containerName="payload-pre-processing" containerID="cri-o://3177974823dfaf7e24e3318b9ba7f793956808d562411b2c082fc35956df6a52" gracePeriod=30
Sep 07 16:34:27.935928 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:27.935901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:27.993380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:27.993350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:28.038686 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.038657    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f12face-2a96-4874-97dc-dde4c22b3017" containerID="3177974823dfaf7e24e3318b9ba7f793956808d562411b2c082fc35956df6a52" exitCode=0
Sep 07 16:34:28.038885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.038735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb" event={"ID":"4f12face-2a96-4874-97dc-dde4c22b3017","Type":"ContainerDied","Data":"3177974823dfaf7e24e3318b9ba7f793956808d562411b2c082fc35956df6a52"}
Sep 07 16:34:28.070475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.070443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:28.132921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.132835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:28.171523 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.171497    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:34:28.278183 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.278143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f12face-2a96-4874-97dc-dde4c22b3017-plugins-config-volume\") pod \"4f12face-2a96-4874-97dc-dde4c22b3017\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") "
Sep 07 16:34:28.278334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.278238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfjk2\" (UniqueName: \"kubernetes.io/projected/4f12face-2a96-4874-97dc-dde4c22b3017-kube-api-access-gfjk2\") pod \"4f12face-2a96-4874-97dc-dde4c22b3017\" (UID: \"4f12face-2a96-4874-97dc-dde4c22b3017\") "
Sep 07 16:34:28.278569 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.278548    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f12face-2a96-4874-97dc-dde4c22b3017-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f12face-2a96-4874-97dc-dde4c22b3017" (UID: "4f12face-2a96-4874-97dc-dde4c22b3017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:34:28.278662 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.278647    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f12face-2a96-4874-97dc-dde4c22b3017-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:28.280537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.280507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f12face-2a96-4874-97dc-dde4c22b3017-kube-api-access-gfjk2" (OuterVolumeSpecName: "kube-api-access-gfjk2") pod "4f12face-2a96-4874-97dc-dde4c22b3017" (UID: "4f12face-2a96-4874-97dc-dde4c22b3017"). InnerVolumeSpecName "kube-api-access-gfjk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:34:28.380251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.380213    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjk2\" (UniqueName: \"kubernetes.io/projected/4f12face-2a96-4874-97dc-dde4c22b3017-kube-api-access-gfjk2\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:28.413982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:28.413949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:29.046098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:29.046066    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"
Sep 07 16:34:29.046098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:29.046093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb" event={"ID":"4f12face-2a96-4874-97dc-dde4c22b3017","Type":"ContainerDied","Data":"ab3f11d1df3a9084f0b9454ae9840337f9211e20de017c38168d3dbddeddc123"}
Sep 07 16:34:29.046617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:29.046158    2577 scope.go:117] "RemoveContainer" containerID="3177974823dfaf7e24e3318b9ba7f793956808d562411b2c082fc35956df6a52"
Sep 07 16:34:29.069267 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:29.069234    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"]
Sep 07 16:34:29.075444 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:29.075416    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cc0983-5dbddff58-5x6qb"]
Sep 07 16:34:29.567883 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:29.567846    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f12face-2a96-4874-97dc-dde4c22b3017" path="/var/lib/kubelet/pods/4f12face-2a96-4874-97dc-dde4c22b3017/volumes"
Sep 07 16:34:32.180878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.180829    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9" podUID="f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" containerName="main" containerID="cri-o://f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686" gracePeriod=45
Sep 07 16:34:32.432366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.432299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:34:32.524217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524189    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-home\") pod \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") "
Sep 07 16:34:32.524398 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524281    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tmp-dir\") pod \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") "
Sep 07 16:34:32.524398 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-model-cache\") pod \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") "
Sep 07 16:34:32.524398 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tls-certs\") pod \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") "
Sep 07 16:34:32.524398 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-245x4\" (UniqueName: \"kubernetes.io/projected/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-kube-api-access-245x4\") pod \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") "
Sep 07 16:34:32.524582 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-dshm\") pod \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\" (UID: \"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4\") "
Sep 07 16:34:32.524582 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-home" (OuterVolumeSpecName: "home") pod "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" (UID: "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:32.524692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" (UID: "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:32.524802 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524778    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:32.524802 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524803    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:32.524991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.524839    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" (UID: "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:32.526873 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.526842    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-kube-api-access-245x4" (OuterVolumeSpecName: "kube-api-access-245x4") pod "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" (UID: "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4"). InnerVolumeSpecName "kube-api-access-245x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:34:32.526873 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.526861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-dshm" (OuterVolumeSpecName: "dshm") pod "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" (UID: "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:32.526996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.526905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" (UID: "f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:34:32.625590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.625556    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:32.625590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.625589    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:32.625836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.625602    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:32.625836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:32.625617    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245x4\" (UniqueName: \"kubernetes.io/projected/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4-kube-api-access-245x4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:33.070234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.070199    2577 generic.go:356] "Generic (PLEG): container finished" podID="f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" containerID="f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686" exitCode=0
Sep 07 16:34:33.070426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.070245    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"
Sep 07 16:34:33.070426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.070290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9" event={"ID":"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4","Type":"ContainerDied","Data":"f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686"}
Sep 07 16:34:33.070426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.070329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9" event={"ID":"f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4","Type":"ContainerDied","Data":"67261a3ef82d105a4b2c63eaf6cd57b003d1ffe1007de0a9ac16c766ab6a1349"}
Sep 07 16:34:33.070426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.070346    2577 scope.go:117] "RemoveContainer" containerID="f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686"
Sep 07 16:34:33.080516 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.080501    2577 scope.go:117] "RemoveContainer" containerID="f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686"
Sep 07 16:34:33.080793 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:34:33.080773    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686\": container with ID starting with f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686 not found: ID does not exist" containerID="f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686"
Sep 07 16:34:33.080879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.080801    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686"} err="failed to get container status \"f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686\": rpc error: code = NotFound desc = could not find container \"f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686\": container with ID starting with f882551cc7b52484c3b40142e07d82d8dce4b4a8168f5a93627f110abc24d686 not found: ID does not exist"
Sep 07 16:34:33.091280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.091255    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"]
Sep 07 16:34:33.093092 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.093075    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-cc0983/explicit-model-cc0983-kserve-ccc894657-dkgx9"]
Sep 07 16:34:33.569405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:33.569372    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" path="/var/lib/kubelet/pods/f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4/volumes"
Sep 07 16:34:34.129406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:34.129368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:34.229910 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:34.229882    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:34.348573 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:34.348531    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"]
Sep 07 16:34:34.348842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:34.348814    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" containerID="cri-o://ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658" gracePeriod=30
Sep 07 16:34:34.368973 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:34.368936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:34.780470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:34.780440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:35.068911 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:35.068828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:35.455514 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:35.455485    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:36.616182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:36.616151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:36.716124 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:36.716083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:36.720128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:36.720094    2577 patch_prober.go:28] interesting pod/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 16:34:36.720262 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:36.720171    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 16:34:36.816127 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:36.816096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:36.916140 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:36.916106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:37.358380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:37.358296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:39.591203 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.591178    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:34:39.697334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-token\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-podinfo\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697383    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-data\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-socket\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697481    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-certs\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istiod-ca-cert\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cl77\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-kube-api-access-9cl77\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-credential-socket\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-envoy\") pod \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\" (UID: \"536bd826-5cfd-4a62-8b3b-a9729f4bc74a\") "
Sep 07 16:34:39.697797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697743    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-data" (OuterVolumeSpecName: "istio-data") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:39.698100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697791    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:39.698100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:39.698100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.697995    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:39.698422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.698142    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.698422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.698160    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.698422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.698172    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.698422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.698185    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.698422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.698208    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:34:39.699818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.699781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:34:39.699934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.699817    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-token" (OuterVolumeSpecName: "istio-token") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:34:39.699934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.699852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:34:39.700008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.699981    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-kube-api-access-9cl77" (OuterVolumeSpecName: "kube-api-access-9cl77") pod "536bd826-5cfd-4a62-8b3b-a9729f4bc74a" (UID: "536bd826-5cfd-4a62-8b3b-a9729f4bc74a"). InnerVolumeSpecName "kube-api-access-9cl77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:34:39.798777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.798714    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.798777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.798747    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.798980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.798789    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.798980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.798802    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:39.798980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:39.798813    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cl77\" (UniqueName: \"kubernetes.io/projected/536bd826-5cfd-4a62-8b3b-a9729f4bc74a-kube-api-access-9cl77\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:34:40.111717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.111625    2577 generic.go:356] "Generic (PLEG): container finished" podID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerID="ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658" exitCode=0
Sep 07 16:34:40.111717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.111677    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"
Sep 07 16:34:40.111717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.111707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" event={"ID":"536bd826-5cfd-4a62-8b3b-a9729f4bc74a","Type":"ContainerDied","Data":"ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658"}
Sep 07 16:34:40.111985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.111773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467" event={"ID":"536bd826-5cfd-4a62-8b3b-a9729f4bc74a","Type":"ContainerDied","Data":"162e7d4b21b359b0f11deecf993a0b1bde6c748c0297fffc252043620f731340"}
Sep 07 16:34:40.111985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.111791    2577 scope.go:117] "RemoveContainer" containerID="ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658"
Sep 07 16:34:40.120907 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.120891    2577 scope.go:117] "RemoveContainer" containerID="1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"
Sep 07 16:34:40.132527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.132508    2577 scope.go:117] "RemoveContainer" containerID="ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658"
Sep 07 16:34:40.132912 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:34:40.132890    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658\": container with ID starting with ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658 not found: ID does not exist" containerID="ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658"
Sep 07 16:34:40.133007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.132921    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658"} err="failed to get container status \"ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658\": rpc error: code = NotFound desc = could not find container \"ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658\": container with ID starting with ebee92ec052f45cf48050a5598cc6de861103e1bab55a1d4e3761b5689a36658 not found: ID does not exist"
Sep 07 16:34:40.133007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.132939    2577 scope.go:117] "RemoveContainer" containerID="1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"
Sep 07 16:34:40.133230 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:34:40.133208    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991\": container with ID starting with 1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991 not found: ID does not exist" containerID="1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"
Sep 07 16:34:40.133289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.133240    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991"} err="failed to get container status \"1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991\": rpc error: code = NotFound desc = could not find container \"1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991\": container with ID starting with 1df03b963081e6e8f9539f6adecb93cdb0e90dfcd6c5f7c65c8a31761a1f8991 not found: ID does not exist"
Sep 07 16:34:40.134795 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.134720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"]
Sep 07 16:34:40.136667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:40.136646    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-cc0983-openshift-default-7dfc59f9b-sq467"]
Sep 07 16:34:41.568851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:41.568815    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" path="/var/lib/kubelet/pods/536bd826-5cfd-4a62-8b3b-a9729f4bc74a/volumes"
Sep 07 16:34:42.914364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:42.914324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:44.437245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:44.437214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:45.435652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:45.435612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:48.130611 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.130525    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"]
Sep 07 16:34:48.131258 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131238    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" containerName="main"
Sep 07 16:34:48.131303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131263    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" containerName="main"
Sep 07 16:34:48.131303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131294    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy"
Sep 07 16:34:48.131369 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131303    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy"
Sep 07 16:34:48.131369 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131332    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f12face-2a96-4874-97dc-dde4c22b3017" containerName="payload-pre-processing"
Sep 07 16:34:48.131369 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131340    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f12face-2a96-4874-97dc-dde4c22b3017" containerName="payload-pre-processing"
Sep 07 16:34:48.131463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131450    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f12face-2a96-4874-97dc-dde4c22b3017" containerName="payload-pre-processing"
Sep 07 16:34:48.131494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131464    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f431e2cd-a2ed-4aef-b5f4-05acc82aaaf4" containerName="main"
Sep 07 16:34:48.131494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131477    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy"
Sep 07 16:34:48.131494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.131488    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy"
Sep 07 16:34:48.136954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.136929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.139739 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.139711    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8a18f8\"/\"kube-root-ca.crt\""
Sep 07 16:34:48.140013 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.139993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"]
Sep 07 16:34:48.140638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.140619    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8a18f8\"/\"openshift-service-ca.crt\""
Sep 07 16:34:48.140730 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.140657    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8a18f8\"/\"default-dockercfg-67ksl\""
Sep 07 16:34:48.140730 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.140678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8a18f8\"/\"noten-model-8a18f8-kserve-self-signed-certs\""
Sep 07 16:34:48.161074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.161045    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:48.184981 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.184946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-home\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.185121 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.184991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-model-cache\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.185121 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.185010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tls-certs\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.185121 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.185027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tmp-dir\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.185121 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.185088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgpd\" (UniqueName: \"kubernetes.io/projected/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-kube-api-access-8hgpd\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.185257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.185150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-dshm\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.285931 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.285895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-home\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.286254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.285949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-model-cache\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.286254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.285972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tls-certs\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.286254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.285995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tmp-dir\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.286254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.286035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgpd\" (UniqueName: \"kubernetes.io/projected/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-kube-api-access-8hgpd\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.286254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.286091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-dshm\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.286822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.286792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-home\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.287043 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.287004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tmp-dir\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.287235 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.287046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-model-cache\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.289288 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.289218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-dshm\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.289660 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.289637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tls-certs\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.293712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.293685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgpd\" (UniqueName: \"kubernetes.io/projected/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-kube-api-access-8hgpd\") pod \"noten-model-8a18f8-kserve-84795d48bc-4446g\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") " pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.450166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.450119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:48.721842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:48.721815    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"]
Sep 07 16:34:48.724432 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:34:48.724397    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a84cf1_dec7_4b9a_8ca9_bbb181a92a7f.slice/crio-5bbb61816aed2c507e907845564ad99a88feacff3a2b9852501d6da97f9841fb WatchSource:0}: Error finding container 5bbb61816aed2c507e907845564ad99a88feacff3a2b9852501d6da97f9841fb: Status 404 returned error can't find the container with id 5bbb61816aed2c507e907845564ad99a88feacff3a2b9852501d6da97f9841fb
Sep 07 16:34:49.163842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:49.163803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g" event={"ID":"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f","Type":"ContainerStarted","Data":"3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c"}
Sep 07 16:34:49.164222 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:49.163855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g" event={"ID":"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f","Type":"ContainerStarted","Data":"5bbb61816aed2c507e907845564ad99a88feacff3a2b9852501d6da97f9841fb"}
Sep 07 16:34:49.179274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:49.179220    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g" podStartSLOduration=1.179205011 podStartE2EDuration="1.179205011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:34:49.178395953 +0000 UTC m=+894.059932921" watchObservedRunningTime="2026-09-07 16:34:49.179205011 +0000 UTC m=+894.060741978"
Sep 07 16:34:51.074324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:51.074284    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:51.174345 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:51.174314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:51.715455 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:51.715399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:52.309625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:52.309590    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:57.449924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:57.449887    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"]
Sep 07 16:34:57.462038 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:57.462014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:57.577432 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:57.577405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:57.794870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:57.794789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:57.855544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:57.855515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:57.917367 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:57.917340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:58.018428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:58.018388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:58.299051 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:58.299017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:34:58.451133 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:58.451102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:34:58.794862 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:34:58.794832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:01.372993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:01.372961    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:01.471462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:01.471423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:02.679779 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:02.679734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:02.780245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:02.780212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:02.981805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:02.981672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:03.081275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:03.081242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:03.226912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:03.226886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:03.325248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:03.325162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:08.218465 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:08.218432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"]
Sep 07 16:35:08.235768 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:08.235719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:09.161976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.161947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:09.266874 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.266841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"]
Sep 07 16:35:09.267280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.267267    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy"
Sep 07 16:35:09.267280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.267281    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="536bd826-5cfd-4a62-8b3b-a9729f4bc74a" containerName="istio-proxy"
Sep 07 16:35:09.271739 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.271724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:09.274672 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.274651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:35:09.277166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.277140    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"]
Sep 07 16:35:09.361533 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.361498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:09.396282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.396248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9c0572-43f3-4296-9937-546b844c131c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph\" (UID: \"bc9c0572-43f3-4296-9937-546b844c131c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:09.497028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.496926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9c0572-43f3-4296-9937-546b844c131c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph\" (UID: \"bc9c0572-43f3-4296-9937-546b844c131c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:09.497597 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.497573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9c0572-43f3-4296-9937-546b844c131c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph\" (UID: \"bc9c0572-43f3-4296-9937-546b844c131c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:09.584705 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.584638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:09.853829 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:09.853712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"]
Sep 07 16:35:09.856565 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:35:09.856532    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9c0572_43f3_4296_9937_546b844c131c.slice/crio-4855956fb276be7e067a8083ea4d90e0b67870f56749b20d318ae46363747723 WatchSource:0}: Error finding container 4855956fb276be7e067a8083ea4d90e0b67870f56749b20d318ae46363747723: Status 404 returned error can't find the container with id 4855956fb276be7e067a8083ea4d90e0b67870f56749b20d318ae46363747723
Sep 07 16:35:10.273808 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:10.273772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph" event={"ID":"bc9c0572-43f3-4296-9937-546b844c131c","Type":"ContainerStarted","Data":"2e4ac1c575bbf29bedf584b515f7e91ea44c73a35ec905a5fb2b19553cabf23a"}
Sep 07 16:35:10.274303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:10.273831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph" event={"ID":"bc9c0572-43f3-4296-9937-546b844c131c","Type":"ContainerStarted","Data":"4855956fb276be7e067a8083ea4d90e0b67870f56749b20d318ae46363747723"}
Sep 07 16:35:10.291293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:10.291241    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph" podStartSLOduration=1.29122767 podStartE2EDuration="1.29122767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:10.2892081 +0000 UTC m=+915.170745069" watchObservedRunningTime="2026-09-07 16:35:10.29122767 +0000 UTC m=+915.172764650"
Sep 07 16:35:11.280970 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:11.280934    2577 generic.go:356] "Generic (PLEG): container finished" podID="bc9c0572-43f3-4296-9937-546b844c131c" containerID="2e4ac1c575bbf29bedf584b515f7e91ea44c73a35ec905a5fb2b19553cabf23a" exitCode=0
Sep 07 16:35:11.281386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:11.281012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph" event={"ID":"bc9c0572-43f3-4296-9937-546b844c131c","Type":"ContainerDied","Data":"2e4ac1c575bbf29bedf584b515f7e91ea44c73a35ec905a5fb2b19553cabf23a"}
Sep 07 16:35:12.415721 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.415696    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:12.470505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.470474    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g" podUID="a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" containerName="main" containerID="cri-o://3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c" gracePeriod=45
Sep 07 16:35:12.527960 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.527936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9c0572-43f3-4296-9937-546b844c131c-maas-api-service-ca\") pod \"bc9c0572-43f3-4296-9937-546b844c131c\" (UID: \"bc9c0572-43f3-4296-9937-546b844c131c\") "
Sep 07 16:35:12.528280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.528255    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9c0572-43f3-4296-9937-546b844c131c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc9c0572-43f3-4296-9937-546b844c131c" (UID: "bc9c0572-43f3-4296-9937-546b844c131c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:12.629452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.629423    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9c0572-43f3-4296-9937-546b844c131c-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:12.718809 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.718785    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:35:12.831547 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-model-cache\") pod \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") "
Sep 07 16:35:12.831691 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-dshm\") pod \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") "
Sep 07 16:35:12.831691 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-home\") pod \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") "
Sep 07 16:35:12.831833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tls-certs\") pod \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") "
Sep 07 16:35:12.831833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tmp-dir\") pod \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") "
Sep 07 16:35:12.831833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831801    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hgpd\" (UniqueName: \"kubernetes.io/projected/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-kube-api-access-8hgpd\") pod \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\" (UID: \"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f\") "
Sep 07 16:35:12.831833 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831802    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" (UID: "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:12.832060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831950    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-home" (OuterVolumeSpecName: "home") pod "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" (UID: "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:12.832060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.831991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" (UID: "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:12.832194 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.832183    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:12.832254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.832196    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:12.832254 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.832206    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:12.834052 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.834024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-dshm" (OuterVolumeSpecName: "dshm") pod "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" (UID: "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:12.834160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.834117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" (UID: "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:35:12.834160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.834120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-kube-api-access-8hgpd" (OuterVolumeSpecName: "kube-api-access-8hgpd") pod "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" (UID: "a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f"). InnerVolumeSpecName "kube-api-access-8hgpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:12.933477 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.933447    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:12.933477 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.933474    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:12.933477 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:12.933486    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgpd\" (UniqueName: \"kubernetes.io/projected/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f-kube-api-access-8hgpd\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:13.292541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.292502    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" containerID="3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c" exitCode=0
Sep 07 16:35:13.292725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.292548    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"
Sep 07 16:35:13.292725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.292597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g" event={"ID":"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f","Type":"ContainerDied","Data":"3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c"}
Sep 07 16:35:13.292725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.292636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g" event={"ID":"a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f","Type":"ContainerDied","Data":"5bbb61816aed2c507e907845564ad99a88feacff3a2b9852501d6da97f9841fb"}
Sep 07 16:35:13.292725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.292652    2577 scope.go:117] "RemoveContainer" containerID="3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c"
Sep 07 16:35:13.294134 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.294111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph" event={"ID":"bc9c0572-43f3-4296-9937-546b844c131c","Type":"ContainerDied","Data":"4855956fb276be7e067a8083ea4d90e0b67870f56749b20d318ae46363747723"}
Sep 07 16:35:13.294134 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.294132    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"
Sep 07 16:35:13.294266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.294157    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4855956fb276be7e067a8083ea4d90e0b67870f56749b20d318ae46363747723"
Sep 07 16:35:13.303385 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.303364    2577 scope.go:117] "RemoveContainer" containerID="3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c"
Sep 07 16:35:13.303796 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:13.303740    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c\": container with ID starting with 3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c not found: ID does not exist" containerID="3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c"
Sep 07 16:35:13.303883 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.303804    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c"} err="failed to get container status \"3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c\": rpc error: code = NotFound desc = could not find container \"3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c\": container with ID starting with 3998b5f4f08e8141d783ff44708c64a56ff5fdfdee0920682a1858df4e70e21c not found: ID does not exist"
Sep 07 16:35:13.316436 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.316409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"]
Sep 07 16:35:13.324092 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.324066    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8a18f8/noten-model-8a18f8-kserve-84795d48bc-4446g"]
Sep 07 16:35:13.568776 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:13.568675    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" path="/var/lib/kubelet/pods/a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f/volumes"
Sep 07 16:35:19.715185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.715149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:19.738707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.738672    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"]
Sep 07 16:35:19.739018 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.738965    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8" podUID="5ce05475-9d94-4808-8cc0-6b462004bf57" containerName="payload-processing" containerID="cri-o://049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4" gracePeriod=30
Sep 07 16:35:19.813058 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.813028    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:19.898927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.898901    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"]
Sep 07 16:35:19.902533 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.902496    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-747d6d-lkcph"]
Sep 07 16:35:19.903643 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.903528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:19.959991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:19.959961    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:20.012322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.012300    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:35:20.112702 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.112674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce05475-9d94-4808-8cc0-6b462004bf57-plugins-config-volume\") pod \"5ce05475-9d94-4808-8cc0-6b462004bf57\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") "
Sep 07 16:35:20.112900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.112830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrqgb\" (UniqueName: \"kubernetes.io/projected/5ce05475-9d94-4808-8cc0-6b462004bf57-kube-api-access-nrqgb\") pod \"5ce05475-9d94-4808-8cc0-6b462004bf57\" (UID: \"5ce05475-9d94-4808-8cc0-6b462004bf57\") "
Sep 07 16:35:20.113092 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.113069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce05475-9d94-4808-8cc0-6b462004bf57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ce05475-9d94-4808-8cc0-6b462004bf57" (UID: "5ce05475-9d94-4808-8cc0-6b462004bf57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:20.115078 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.115048    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce05475-9d94-4808-8cc0-6b462004bf57-kube-api-access-nrqgb" (OuterVolumeSpecName: "kube-api-access-nrqgb") pod "5ce05475-9d94-4808-8cc0-6b462004bf57" (UID: "5ce05475-9d94-4808-8cc0-6b462004bf57"). InnerVolumeSpecName "kube-api-access-nrqgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:20.214334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.214290    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce05475-9d94-4808-8cc0-6b462004bf57-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:20.214334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.214325    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqgb\" (UniqueName: \"kubernetes.io/projected/5ce05475-9d94-4808-8cc0-6b462004bf57-kube-api-access-nrqgb\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:20.238677 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.238598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:20.336116 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.336080    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ce05475-9d94-4808-8cc0-6b462004bf57" containerID="049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4" exitCode=0
Sep 07 16:35:20.336116 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.336120    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"
Sep 07 16:35:20.336353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.336170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8" event={"ID":"5ce05475-9d94-4808-8cc0-6b462004bf57","Type":"ContainerDied","Data":"049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4"}
Sep 07 16:35:20.336353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.336249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8" event={"ID":"5ce05475-9d94-4808-8cc0-6b462004bf57","Type":"ContainerDied","Data":"a769911f789b73343c66d104a0eb8bb554cb0a14ad6bb9e1bde411c338750ef6"}
Sep 07 16:35:20.336353 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.336274    2577 scope.go:117] "RemoveContainer" containerID="049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4"
Sep 07 16:35:20.345852 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.345834    2577 scope.go:117] "RemoveContainer" containerID="049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4"
Sep 07 16:35:20.346101 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:20.346083    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4\": container with ID starting with 049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4 not found: ID does not exist" containerID="049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4"
Sep 07 16:35:20.346156 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.346110    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4"} err="failed to get container status \"049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4\": rpc error: code = NotFound desc = could not find container \"049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4\": container with ID starting with 049ac66079a3ad51ab62e867af8b6d18a2e763344cf35cccc9105ca0fa230df4 not found: ID does not exist"
Sep 07 16:35:20.357704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.357678    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"]
Sep 07 16:35:20.361348 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:20.361330    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-747d6d-5564846fb5-qxgb8"]
Sep 07 16:35:21.568426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:21.568372    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ce05475-9d94-4808-8cc0-6b462004bf57" path="/var/lib/kubelet/pods/5ce05475-9d94-4808-8cc0-6b462004bf57/volumes"
Sep 07 16:35:21.568813 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:21.568781    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc9c0572-43f3-4296-9937-546b844c131c" path="/var/lib/kubelet/pods/bc9c0572-43f3-4296-9937-546b844c131c/volumes"
Sep 07 16:35:23.239476 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.239438    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" podUID="25ca7402-f7c4-47f3-a025-0cdc03538402" containerName="main" containerID="cri-o://442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a" gracePeriod=45
Sep 07 16:35:23.498591 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.498539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:35:23.545588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.545555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpf7b\" (UniqueName: \"kubernetes.io/projected/25ca7402-f7c4-47f3-a025-0cdc03538402-kube-api-access-gpf7b\") pod \"25ca7402-f7c4-47f3-a025-0cdc03538402\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") "
Sep 07 16:35:23.545775 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.545595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ca7402-f7c4-47f3-a025-0cdc03538402-tls-certs\") pod \"25ca7402-f7c4-47f3-a025-0cdc03538402\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") "
Sep 07 16:35:23.545860 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.545790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-model-cache\") pod \"25ca7402-f7c4-47f3-a025-0cdc03538402\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") "
Sep 07 16:35:23.545860 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.545831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-tmp-dir\") pod \"25ca7402-f7c4-47f3-a025-0cdc03538402\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") "
Sep 07 16:35:23.545860 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.545856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-dshm\") pod \"25ca7402-f7c4-47f3-a025-0cdc03538402\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") "
Sep 07 16:35:23.546018 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.545912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-home\") pod \"25ca7402-f7c4-47f3-a025-0cdc03538402\" (UID: \"25ca7402-f7c4-47f3-a025-0cdc03538402\") "
Sep 07 16:35:23.546105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.546080    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25ca7402-f7c4-47f3-a025-0cdc03538402" (UID: "25ca7402-f7c4-47f3-a025-0cdc03538402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:23.546185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.546097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-model-cache" (OuterVolumeSpecName: "model-cache") pod "25ca7402-f7c4-47f3-a025-0cdc03538402" (UID: "25ca7402-f7c4-47f3-a025-0cdc03538402"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:23.546272 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.546247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-home" (OuterVolumeSpecName: "home") pod "25ca7402-f7c4-47f3-a025-0cdc03538402" (UID: "25ca7402-f7c4-47f3-a025-0cdc03538402"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:23.546396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.546342    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:23.546396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.546378    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:23.546396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.546390    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:23.548626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.548593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ca7402-f7c4-47f3-a025-0cdc03538402-kube-api-access-gpf7b" (OuterVolumeSpecName: "kube-api-access-gpf7b") pod "25ca7402-f7c4-47f3-a025-0cdc03538402" (UID: "25ca7402-f7c4-47f3-a025-0cdc03538402"). InnerVolumeSpecName "kube-api-access-gpf7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:23.548818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.548675    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ca7402-f7c4-47f3-a025-0cdc03538402-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25ca7402-f7c4-47f3-a025-0cdc03538402" (UID: "25ca7402-f7c4-47f3-a025-0cdc03538402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:35:23.548818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.548776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-dshm" (OuterVolumeSpecName: "dshm") pod "25ca7402-f7c4-47f3-a025-0cdc03538402" (UID: "25ca7402-f7c4-47f3-a025-0cdc03538402"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:23.623744 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.623707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:23.647242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.647185    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ca7402-f7c4-47f3-a025-0cdc03538402-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:23.647242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.647216    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpf7b\" (UniqueName: \"kubernetes.io/projected/25ca7402-f7c4-47f3-a025-0cdc03538402-kube-api-access-gpf7b\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:23.647242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:23.647232    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ca7402-f7c4-47f3-a025-0cdc03538402-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:24.360369 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.360331    2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca7402-f7c4-47f3-a025-0cdc03538402" containerID="442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a" exitCode=0
Sep 07 16:35:24.361002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.360409    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"
Sep 07 16:35:24.361002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.360441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" event={"ID":"25ca7402-f7c4-47f3-a025-0cdc03538402","Type":"ContainerDied","Data":"442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a"}
Sep 07 16:35:24.361002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.360481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" event={"ID":"25ca7402-f7c4-47f3-a025-0cdc03538402","Type":"ContainerDied","Data":"5c209e5658c985c1916ed709112ef820b29a2ed41a39f2e0065f05891148781c"}
Sep 07 16:35:24.361002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.360499    2577 scope.go:117] "RemoveContainer" containerID="442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a"
Sep 07 16:35:24.369442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.369425    2577 scope.go:117] "RemoveContainer" containerID="442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a"
Sep 07 16:35:24.369708 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:24.369692    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a\": container with ID starting with 442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a not found: ID does not exist" containerID="442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a"
Sep 07 16:35:24.369771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.369716    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a"} err="failed to get container status \"442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a\": rpc error: code = NotFound desc = could not find container \"442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a\": container with ID starting with 442723779bed101e4ff8430fbe018e310777bc2434d1b8da87b3e6cecf67441a not found: ID does not exist"
Sep 07 16:35:24.376494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.376473    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"]
Sep 07 16:35:24.382663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:24.382641    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r"]
Sep 07 16:35:25.567729 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:25.567689    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ca7402-f7c4-47f3-a025-0cdc03538402" path="/var/lib/kubelet/pods/25ca7402-f7c4-47f3-a025-0cdc03538402/volumes"
Sep 07 16:35:25.928693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:25.928664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:26.028747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.028715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:26.146197 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.146164    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"]
Sep 07 16:35:26.146423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.146390    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy" containerID="cri-o://2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1" gracePeriod=30
Sep 07 16:35:26.170274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.170224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:26.293786 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.293675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:26.512575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.512543    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"]
Sep 07 16:35:26.512975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.512962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" containerName="main"
Sep 07 16:35:26.513022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.512976    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" containerName="main"
Sep 07 16:35:26.513022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.512986    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ca7402-f7c4-47f3-a025-0cdc03538402" containerName="main"
Sep 07 16:35:26.513022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.512991    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ca7402-f7c4-47f3-a025-0cdc03538402" containerName="main"
Sep 07 16:35:26.513022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513005    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce05475-9d94-4808-8cc0-6b462004bf57" containerName="payload-processing"
Sep 07 16:35:26.513022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513011    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce05475-9d94-4808-8cc0-6b462004bf57" containerName="payload-processing"
Sep 07 16:35:26.513172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513029    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9c0572-43f3-4296-9937-546b844c131c" containerName="revoke-keys"
Sep 07 16:35:26.513172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513034    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9c0572-43f3-4296-9937-546b844c131c" containerName="revoke-keys"
Sep 07 16:35:26.513172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513091    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce05475-9d94-4808-8cc0-6b462004bf57" containerName="payload-processing"
Sep 07 16:35:26.513172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513100    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a84cf1-dec7-4b9a-8ca9-bbb181a92a7f" containerName="main"
Sep 07 16:35:26.513172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513107    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ca7402-f7c4-47f3-a025-0cdc03538402" containerName="main"
Sep 07 16:35:26.513172 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.513114    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9c0572-43f3-4296-9937-546b844c131c" containerName="revoke-keys"
Sep 07 16:35:26.517770 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.517735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:26.521442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.521422    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:35:26.523691 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.523667    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"]
Sep 07 16:35:26.578040 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.577955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46\" (UID: \"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:26.679215 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.679178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46\" (UID: \"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:26.679800 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.679745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46\" (UID: \"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:26.759259 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.759228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:26.830338 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.830252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:26.932367 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:26.932330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:27.032062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:27.032025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:27.099200 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:27.099173    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"]
Sep 07 16:35:27.100682 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:35:27.100659    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95dd58e_727b_42ee_b151_dbb9ddd1e0a4.slice/crio-35cc95919f38552687cd18aaed2fa48392016ee7a649b32531def56b48c1d5bb WatchSource:0}: Error finding container 35cc95919f38552687cd18aaed2fa48392016ee7a649b32531def56b48c1d5bb: Status 404 returned error can't find the container with id 35cc95919f38552687cd18aaed2fa48392016ee7a649b32531def56b48c1d5bb
Sep 07 16:35:27.335423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:27.335393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:27.378880 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:27.378789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46" event={"ID":"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4","Type":"ContainerStarted","Data":"976f37fd9e6264beae432028e61dc26a7d4e3ec371871c2238a53657b8b4cae2"}
Sep 07 16:35:27.378880 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:27.378853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46" event={"ID":"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4","Type":"ContainerStarted","Data":"35cc95919f38552687cd18aaed2fa48392016ee7a649b32531def56b48c1d5bb"}
Sep 07 16:35:27.394761 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:27.394705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46" podStartSLOduration=1.394692577 podStartE2EDuration="1.394692577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:27.392829766 +0000 UTC m=+932.274366736" watchObservedRunningTime="2026-09-07 16:35:27.394692577 +0000 UTC m=+932.276229544"
Sep 07 16:35:28.392246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:28.392210    2577 generic.go:356] "Generic (PLEG): container finished" podID="e95dd58e-727b-42ee-b151-dbb9ddd1e0a4" containerID="976f37fd9e6264beae432028e61dc26a7d4e3ec371871c2238a53657b8b4cae2" exitCode=0
Sep 07 16:35:28.392650 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:28.392295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46" event={"ID":"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4","Type":"ContainerDied","Data":"976f37fd9e6264beae432028e61dc26a7d4e3ec371871c2238a53657b8b4cae2"}
Sep 07 16:35:28.485638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:28.485603    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-747d6d/test-model-747d6d-kserve-fd78bf465-jvk9r" podUID="25ca7402-f7c4-47f3-a025-0cdc03538402" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 07 16:35:28.540192 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:28.540105    2577 event.go:359] "Server rejected event (will not retry!)" err="namespaces \"ai-tenant-e2e-inf-a-747d6d\" not found" event="&Event{ObjectMeta:{test-model-747d6d-kserve-fd78bf465-jvk9r.18d31725e2fc6dcf  ai-tenant-e2e-inf-a-747d6d    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:ai-tenant-e2e-inf-a-747d6d,Name:test-model-747d6d-kserve-fd78bf465-jvk9r,UID:25ca7402-f7c4-47f3-a025-0cdc03538402,APIVersion:v1,ResourceVersion:30417,FieldPath:spec.containers{main},},Reason:Unhealthy,Message:Readiness probe failed: Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers),Source:EventSource{Component:kubelet,Host:ip-10-0-129-87.ec2.internal,},FirstTimestamp:2026-09-07 16:35:28.485641679 +0000 UTC m=+933.367178648,LastTimestamp:2026-09-07 16:35:28.485641679 +0000 UTC m=+933.367178648,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-87.ec2.internal,}"
Sep 07 16:35:29.540812 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:29.540788    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:29.607356 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:29.607327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4-maas-api-service-ca\") pod \"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4\" (UID: \"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4\") "
Sep 07 16:35:29.607712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:29.607688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e95dd58e-727b-42ee-b151-dbb9ddd1e0a4" (UID: "e95dd58e-727b-42ee-b151-dbb9ddd1e0a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:29.708909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:29.708821    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:30.403693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:30.403666    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"
Sep 07 16:35:30.403693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:30.403678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46" event={"ID":"e95dd58e-727b-42ee-b151-dbb9ddd1e0a4","Type":"ContainerDied","Data":"35cc95919f38552687cd18aaed2fa48392016ee7a649b32531def56b48c1d5bb"}
Sep 07 16:35:30.403924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:30.403728    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35cc95919f38552687cd18aaed2fa48392016ee7a649b32531def56b48c1d5bb"
Sep 07 16:35:31.403768 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.403728    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:35:31.409792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.409709    2577 generic.go:356] "Generic (PLEG): container finished" podID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerID="2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1" exitCode=0
Sep 07 16:35:31.409792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.409774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"
Sep 07 16:35:31.409948 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.409793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" event={"ID":"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f","Type":"ContainerDied","Data":"2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1"}
Sep 07 16:35:31.409948 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.409852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp" event={"ID":"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f","Type":"ContainerDied","Data":"327a972ab4b28cad6cf950f9b6814dbf88f6fe6cdc7c955890a8736d10d836bc"}
Sep 07 16:35:31.409948 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.409878    2577 scope.go:117] "RemoveContainer" containerID="2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1"
Sep 07 16:35:31.420314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.420295    2577 scope.go:117] "RemoveContainer" containerID="211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"
Sep 07 16:35:31.431301 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.431275    2577 scope.go:117] "RemoveContainer" containerID="2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1"
Sep 07 16:35:31.431700 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:31.431680    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1\": container with ID starting with 2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1 not found: ID does not exist" containerID="2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1"
Sep 07 16:35:31.431793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.431713    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1"} err="failed to get container status \"2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1\": rpc error: code = NotFound desc = could not find container \"2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1\": container with ID starting with 2a47c452df07d174d6efe6c8fb0da2b481cdff692feee6e95e5fbcc550cdd2d1 not found: ID does not exist"
Sep 07 16:35:31.431793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.431738    2577 scope.go:117] "RemoveContainer" containerID="211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"
Sep 07 16:35:31.432267 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:31.432249    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21\": container with ID starting with 211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21 not found: ID does not exist" containerID="211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"
Sep 07 16:35:31.432342 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.432272    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21"} err="failed to get container status \"211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21\": rpc error: code = NotFound desc = could not find container \"211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21\": container with ID starting with 211216c057da99bc41de25d44171c0180a318652df55361af5a6c5160f913a21 not found: ID does not exist"
Sep 07 16:35:31.528098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528055    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-envoy\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528145    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istiod-ca-cert\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528166    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-socket\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-podinfo\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-token\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528476 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-data\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528476 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7kk5\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-kube-api-access-r7kk5\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-certs\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-credential-socket\") pod \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\" (UID: \"3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f\") "
Sep 07 16:35:31.528584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:31.528774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:31.528827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:31.528827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:31.529280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528999    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.529280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.528999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:31.529280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.529019    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.529280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.529032    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.529280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.529045    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.530693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.530666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:35:31.530815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.530702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:31.530861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.530814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-kube-api-access-r7kk5" (OuterVolumeSpecName: "kube-api-access-r7kk5") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "kube-api-access-r7kk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:31.530939 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.530926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" (UID: "3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:31.629925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.629890    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.629925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.629923    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.630125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.629936    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7kk5\" (UniqueName: \"kubernetes.io/projected/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-kube-api-access-r7kk5\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.630125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.629950    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.630125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.629964    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:31.726087 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.726006    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"]
Sep 07 16:35:31.730229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:31.730204    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-747d6d-openshift-default-598bd45998-j9bfp"]
Sep 07 16:35:33.245032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.245000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:33.345659 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.345621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:33.568034 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.567956    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" path="/var/lib/kubelet/pods/3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f/volumes"
Sep 07 16:35:33.582365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582336    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"]
Sep 07 16:35:33.582712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582701    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy"
Sep 07 16:35:33.582792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy"
Sep 07 16:35:33.582792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582721    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95dd58e-727b-42ee-b151-dbb9ddd1e0a4" containerName="revoke-keys"
Sep 07 16:35:33.582792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582726    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95dd58e-727b-42ee-b151-dbb9ddd1e0a4" containerName="revoke-keys"
Sep 07 16:35:33.582792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582736    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy"
Sep 07 16:35:33.582792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582741    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy"
Sep 07 16:35:33.582954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582813    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy"
Sep 07 16:35:33.582954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582822    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95dd58e-727b-42ee-b151-dbb9ddd1e0a4" containerName="revoke-keys"
Sep 07 16:35:33.582954 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.582831    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa2fc3f-9001-4f0b-8e0e-5aad52aba62f" containerName="istio-proxy"
Sep 07 16:35:33.587532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.587513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:33.590269 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.590247    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:35:33.594257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.594232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"]
Sep 07 16:35:33.647727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.647693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d170bd39-b7e8-440b-8522-7c54ccb2b426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx\" (UID: \"d170bd39-b7e8-440b-8522-7c54ccb2b426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:33.748932 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.748888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d170bd39-b7e8-440b-8522-7c54ccb2b426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx\" (UID: \"d170bd39-b7e8-440b-8522-7c54ccb2b426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:33.749550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.749527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d170bd39-b7e8-440b-8522-7c54ccb2b426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx\" (UID: \"d170bd39-b7e8-440b-8522-7c54ccb2b426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:33.761935 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.761899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:33.900145 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:33.900111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:34.177002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:34.176975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"]
Sep 07 16:35:34.178864 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:35:34.178839    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd170bd39_b7e8_440b_8522_7c54ccb2b426.slice/crio-3b67b452ba2b4d1ab90ca30986438b8d5a475dc68c6ba8d1b3532f0cb3dc6aec WatchSource:0}: Error finding container 3b67b452ba2b4d1ab90ca30986438b8d5a475dc68c6ba8d1b3532f0cb3dc6aec: Status 404 returned error can't find the container with id 3b67b452ba2b4d1ab90ca30986438b8d5a475dc68c6ba8d1b3532f0cb3dc6aec
Sep 07 16:35:34.428439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:34.428335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx" event={"ID":"d170bd39-b7e8-440b-8522-7c54ccb2b426","Type":"ContainerStarted","Data":"3b67b452ba2b4d1ab90ca30986438b8d5a475dc68c6ba8d1b3532f0cb3dc6aec"}
Sep 07 16:35:35.435548 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:35.435510    2577 generic.go:356] "Generic (PLEG): container finished" podID="d170bd39-b7e8-440b-8522-7c54ccb2b426" containerID="1bfda06fae85ffae84a47210895c3c238b5280bb39f13ca35d613f09f0a77887" exitCode=0
Sep 07 16:35:35.436011 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:35.435573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx" event={"ID":"d170bd39-b7e8-440b-8522-7c54ccb2b426","Type":"ContainerDied","Data":"1bfda06fae85ffae84a47210895c3c238b5280bb39f13ca35d613f09f0a77887"}
Sep 07 16:35:36.575152 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:36.575128    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:36.640983 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:36.640946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:36.676099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:36.676059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d170bd39-b7e8-440b-8522-7c54ccb2b426-maas-api-service-ca\") pod \"d170bd39-b7e8-440b-8522-7c54ccb2b426\" (UID: \"d170bd39-b7e8-440b-8522-7c54ccb2b426\") "
Sep 07 16:35:36.676366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:36.676330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d170bd39-b7e8-440b-8522-7c54ccb2b426-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d170bd39-b7e8-440b-8522-7c54ccb2b426" (UID: "d170bd39-b7e8-440b-8522-7c54ccb2b426"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:36.777575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:36.777468    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d170bd39-b7e8-440b-8522-7c54ccb2b426-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:37.197182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.197158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:37.228132 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.228102    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"]
Sep 07 16:35:37.228336 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.228312    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr" podUID="021a428a-b211-454b-b1ef-0235e1b939f2" containerName="payload-processing" containerID="cri-o://fb7a06a7d0961cfb449a46888c5f4a183484662b0c1941a3936ec0237ad85ffd" gracePeriod=30
Sep 07 16:35:37.235424 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.235393    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"]
Sep 07 16:35:37.296902 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.296870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:37.375455 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.375426    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"]
Sep 07 16:35:37.377351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.377327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:37.377449 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.377347    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7d399-c9l46"]
Sep 07 16:35:37.433237 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.433192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:37.447662 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.447595    2577 generic.go:356] "Generic (PLEG): container finished" podID="021a428a-b211-454b-b1ef-0235e1b939f2" containerID="fb7a06a7d0961cfb449a46888c5f4a183484662b0c1941a3936ec0237ad85ffd" exitCode=0
Sep 07 16:35:37.447806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.447667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr" event={"ID":"021a428a-b211-454b-b1ef-0235e1b939f2","Type":"ContainerDied","Data":"fb7a06a7d0961cfb449a46888c5f4a183484662b0c1941a3936ec0237ad85ffd"}
Sep 07 16:35:37.449045 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.449023    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"
Sep 07 16:35:37.449178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.449054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx" event={"ID":"d170bd39-b7e8-440b-8522-7c54ccb2b426","Type":"ContainerDied","Data":"3b67b452ba2b4d1ab90ca30986438b8d5a475dc68c6ba8d1b3532f0cb3dc6aec"}
Sep 07 16:35:37.449178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.449090    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b67b452ba2b4d1ab90ca30986438b8d5a475dc68c6ba8d1b3532f0cb3dc6aec"
Sep 07 16:35:37.449305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.449275    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q" podUID="287cc9a7-7bc1-45d6-b9d1-869761a7c63f" containerName="payload-pre-processing" containerID="cri-o://135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854" gracePeriod=30
Sep 07 16:35:37.569432 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.569402    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95dd58e-727b-42ee-b151-dbb9ddd1e0a4" path="/var/lib/kubelet/pods/e95dd58e-727b-42ee-b151-dbb9ddd1e0a4/volumes"
Sep 07 16:35:37.577622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.577593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:35:37.686521 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.686486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnr5k\" (UniqueName: \"kubernetes.io/projected/021a428a-b211-454b-b1ef-0235e1b939f2-kube-api-access-vnr5k\") pod \"021a428a-b211-454b-b1ef-0235e1b939f2\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") "
Sep 07 16:35:37.686689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.686592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021a428a-b211-454b-b1ef-0235e1b939f2-plugins-config-volume\") pod \"021a428a-b211-454b-b1ef-0235e1b939f2\" (UID: \"021a428a-b211-454b-b1ef-0235e1b939f2\") "
Sep 07 16:35:37.687005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.686957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021a428a-b211-454b-b1ef-0235e1b939f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "021a428a-b211-454b-b1ef-0235e1b939f2" (UID: "021a428a-b211-454b-b1ef-0235e1b939f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:37.689088 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.689064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021a428a-b211-454b-b1ef-0235e1b939f2-kube-api-access-vnr5k" (OuterVolumeSpecName: "kube-api-access-vnr5k") pod "021a428a-b211-454b-b1ef-0235e1b939f2" (UID: "021a428a-b211-454b-b1ef-0235e1b939f2"). InnerVolumeSpecName "kube-api-access-vnr5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:37.706879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.706814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:37.709666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.709649    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:35:37.788125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.788099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8zxw\" (UniqueName: \"kubernetes.io/projected/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-kube-api-access-k8zxw\") pod \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") "
Sep 07 16:35:37.788266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.788156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-plugins-config-volume\") pod \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\" (UID: \"287cc9a7-7bc1-45d6-b9d1-869761a7c63f\") "
Sep 07 16:35:37.788396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.788366    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnr5k\" (UniqueName: \"kubernetes.io/projected/021a428a-b211-454b-b1ef-0235e1b939f2-kube-api-access-vnr5k\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:37.788433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.788402    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021a428a-b211-454b-b1ef-0235e1b939f2-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:37.788510 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.788489    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "287cc9a7-7bc1-45d6-b9d1-869761a7c63f" (UID: "287cc9a7-7bc1-45d6-b9d1-869761a7c63f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:37.790453 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.790434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-kube-api-access-k8zxw" (OuterVolumeSpecName: "kube-api-access-k8zxw") pod "287cc9a7-7bc1-45d6-b9d1-869761a7c63f" (UID: "287cc9a7-7bc1-45d6-b9d1-869761a7c63f"). InnerVolumeSpecName "kube-api-access-k8zxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:37.889328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.889296    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8zxw\" (UniqueName: \"kubernetes.io/projected/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-kube-api-access-k8zxw\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:37.889328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:37.889321    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287cc9a7-7bc1-45d6-b9d1-869761a7c63f-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:38.455561 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.455529    2577 generic.go:356] "Generic (PLEG): container finished" podID="287cc9a7-7bc1-45d6-b9d1-869761a7c63f" containerID="135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854" exitCode=0
Sep 07 16:35:38.455778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.455569    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"
Sep 07 16:35:38.455778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.455615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q" event={"ID":"287cc9a7-7bc1-45d6-b9d1-869761a7c63f","Type":"ContainerDied","Data":"135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854"}
Sep 07 16:35:38.455778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.455661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q" event={"ID":"287cc9a7-7bc1-45d6-b9d1-869761a7c63f","Type":"ContainerDied","Data":"229aed66c0a4bb0abda5b3b70c20b8cb30f7eb22ee13b8dccd511388c65986e9"}
Sep 07 16:35:38.455778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.455684    2577 scope.go:117] "RemoveContainer" containerID="135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854"
Sep 07 16:35:38.457111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.456953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"
Sep 07 16:35:38.457111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.456991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr" event={"ID":"021a428a-b211-454b-b1ef-0235e1b939f2","Type":"ContainerDied","Data":"5b4489cec29dd96478555b085a530e3efb9f8631128f0e0f05adc5cc708b228b"}
Sep 07 16:35:38.464575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.464550    2577 scope.go:117] "RemoveContainer" containerID="135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854"
Sep 07 16:35:38.464879 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:38.464859    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854\": container with ID starting with 135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854 not found: ID does not exist" containerID="135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854"
Sep 07 16:35:38.464938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.464887    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854"} err="failed to get container status \"135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854\": rpc error: code = NotFound desc = could not find container \"135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854\": container with ID starting with 135749937ba15f963403ed6549289df67732354ccc9fd3521b5ae1525419e854 not found: ID does not exist"
Sep 07 16:35:38.464938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.464904    2577 scope.go:117] "RemoveContainer" containerID="fb7a06a7d0961cfb449a46888c5f4a183484662b0c1941a3936ec0237ad85ffd"
Sep 07 16:35:38.480104 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.480080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"]
Sep 07 16:35:38.482021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.482005    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7d399-5fc5bb6fc8-csd7q"]
Sep 07 16:35:38.491456 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.491439    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"]
Sep 07 16:35:38.494524 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:38.494503    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a7d399-577d9b4c6f-8rblr"]
Sep 07 16:35:39.567331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:39.567297    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="021a428a-b211-454b-b1ef-0235e1b939f2" path="/var/lib/kubelet/pods/021a428a-b211-454b-b1ef-0235e1b939f2/volumes"
Sep 07 16:35:39.567677 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:39.567641    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="287cc9a7-7bc1-45d6-b9d1-869761a7c63f" path="/var/lib/kubelet/pods/287cc9a7-7bc1-45d6-b9d1-869761a7c63f/volumes"
Sep 07 16:35:39.806796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:39.806738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:43.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:43.363968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:43.464207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:43.464173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:43.615295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:43.615209    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"]
Sep 07 16:35:43.615442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:43.615408    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" containerID="cri-o://dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4" gracePeriod=30
Sep 07 16:35:43.685097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:43.685061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:43.803198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:43.803167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.136603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.136571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.398562 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.398479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.412689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.412660    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"]
Sep 07 16:35:44.413130 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413109    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021a428a-b211-454b-b1ef-0235e1b939f2" containerName="payload-processing"
Sep 07 16:35:44.413130 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413128    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="021a428a-b211-454b-b1ef-0235e1b939f2" containerName="payload-processing"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413142    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287cc9a7-7bc1-45d6-b9d1-869761a7c63f" containerName="payload-pre-processing"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="287cc9a7-7bc1-45d6-b9d1-869761a7c63f" containerName="payload-pre-processing"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413186    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d170bd39-b7e8-440b-8522-7c54ccb2b426" containerName="revoke-keys"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413196    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d170bd39-b7e8-440b-8522-7c54ccb2b426" containerName="revoke-keys"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413275    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="021a428a-b211-454b-b1ef-0235e1b939f2" containerName="payload-processing"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413287    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="287cc9a7-7bc1-45d6-b9d1-869761a7c63f" containerName="payload-pre-processing"
Sep 07 16:35:44.413329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.413294    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d170bd39-b7e8-440b-8522-7c54ccb2b426" containerName="revoke-keys"
Sep 07 16:35:44.418107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.418088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.421804 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.421786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-8d59e2-openshift-default-dockercfg-75zrl\""
Sep 07 16:35:44.430924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.430904    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"]
Sep 07 16:35:44.479289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.479256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.542164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.542130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.552448 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf75\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-kube-api-access-6xf75\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.552624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-credential-socket\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.552624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-podinfo\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.552624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-envoy\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.552857 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-socket\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.552857 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-data\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.552857 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-token\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.553003 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-certs\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.553003 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.552892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bffe79-62d9-4cbf-864d-fbe1067631eb-istiod-ca-cert\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.618720 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.618684    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"]
Sep 07 16:35:44.622970 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.622924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.625089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.625070    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65f07c0f-52qwx"]
Sep 07 16:35:44.654284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-certs\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bffe79-62d9-4cbf-864d-fbe1067631eb-istiod-ca-cert\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf75\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-kube-api-access-6xf75\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-credential-socket\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-podinfo\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-envoy\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-socket\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-data\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654537 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-token\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-certs\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.654887 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-credential-socket\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.655016 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.654995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-data\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.655111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.655033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-socket\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.655270 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.655250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bffe79-62d9-4cbf-864d-fbe1067631eb-istiod-ca-cert\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.656891 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.656871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-envoy\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.657026 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.657011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-podinfo\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.662246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.662218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-token\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.662454 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.662439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf75\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-kube-api-access-6xf75\") pod \"e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") " pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.684656 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.684621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:44.730293 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.730263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:44.969689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:44.969593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:45.011179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.011156    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"]
Sep 07 16:35:45.013415 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:35:45.013387    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bffe79_62d9_4cbf_864d_fbe1067631eb.slice/crio-a8d6731457e5784d52a7e5127796046324ea24e0a897142bf8b624ccd895c8a6 WatchSource:0}: Error finding container a8d6731457e5784d52a7e5127796046324ea24e0a897142bf8b624ccd895c8a6: Status 404 returned error can't find the container with id a8d6731457e5784d52a7e5127796046324ea24e0a897142bf8b624ccd895c8a6
Sep 07 16:35:45.015996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.015960    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:35:45.016111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.016029    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:35:45.016111 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.016057    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:35:45.069469 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.069429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:45.496992 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.496955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" event={"ID":"89bffe79-62d9-4cbf-864d-fbe1067631eb","Type":"ContainerStarted","Data":"dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"}
Sep 07 16:35:45.497393 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.497014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" event={"ID":"89bffe79-62d9-4cbf-864d-fbe1067631eb","Type":"ContainerStarted","Data":"a8d6731457e5784d52a7e5127796046324ea24e0a897142bf8b624ccd895c8a6"}
Sep 07 16:35:45.518059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.518003    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podStartSLOduration=1.517982021 podStartE2EDuration="1.517982021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:45.517624252 +0000 UTC m=+950.399161221" watchObservedRunningTime="2026-09-07 16:35:45.517982021 +0000 UTC m=+950.399518991"
Sep 07 16:35:45.568430 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.568390    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d170bd39-b7e8-440b-8522-7c54ccb2b426" path="/var/lib/kubelet/pods/d170bd39-b7e8-440b-8522-7c54ccb2b426/volumes"
Sep 07 16:35:45.730797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:45.730748    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:46.615843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:46.615803    2577 patch_prober.go:28] interesting pod/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 16:35:46.616216 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:46.615860    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 16:35:46.735818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:46.735790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:47.373583 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:47.373552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:47.472598 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:47.472562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:47.507602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:47.507572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:47.508729 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:47.508709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:35:48.691821 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.691788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:48.822903 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.822865    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:48.873675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.873652    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:35:48.918340 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.918316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:48.998529 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-certs\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-socket\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-credential-socket\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998640    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb82f\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-kube-api-access-qb82f\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-data\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1fab42-95c4-4eff-b873-0beba327abea-istiod-ca-cert\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998700    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1fab42-95c4-4eff-b873-0beba327abea-istio-podinfo\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.998725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-envoy\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.999089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:48.999089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-istio-token\") pod \"3d1fab42-95c4-4eff-b873-0beba327abea\" (UID: \"3d1fab42-95c4-4eff-b873-0beba327abea\") "
Sep 07 16:35:48.999089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.998901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:48.999089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:48.999089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999018    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:48.999316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999129    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1fab42-95c4-4eff-b873-0beba327abea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:35:48.999316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999214    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:48.999316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999231    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:48.999316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999241    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1fab42-95c4-4eff-b873-0beba327abea-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:48.999316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999248    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:48.999316 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:48.999255    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:49.001063 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.001038    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d1fab42-95c4-4eff-b873-0beba327abea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:35:49.001176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.001145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:35:49.001226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.001188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:49.001226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.001207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-kube-api-access-qb82f" (OuterVolumeSpecName: "kube-api-access-qb82f") pod "3d1fab42-95c4-4eff-b873-0beba327abea" (UID: "3d1fab42-95c4-4eff-b873-0beba327abea"). InnerVolumeSpecName "kube-api-access-qb82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:35:49.100474 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.100435    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb82f\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-kube-api-access-qb82f\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:49.100474 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.100465    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1fab42-95c4-4eff-b873-0beba327abea-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:49.100474 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.100474    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1fab42-95c4-4eff-b873-0beba327abea-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:49.100474 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.100482    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1fab42-95c4-4eff-b873-0beba327abea-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:35:49.242619 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.242539    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:49.340727 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.340691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:49.341846 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.341825    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"]
Sep 07 16:35:49.342286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.342272    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy"
Sep 07 16:35:49.342328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.342288    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy"
Sep 07 16:35:49.342328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.342320    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy"
Sep 07 16:35:49.342328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.342326    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy"
Sep 07 16:35:49.342461 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.342403    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy"
Sep 07 16:35:49.342461 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.342414    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" containerName="istio-proxy"
Sep 07 16:35:49.345944 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.345927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.348405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.348385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8d59e2\""
Sep 07 16:35:49.348531 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.348514    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8d59e2-dockercfg-gtpc2\""
Sep 07 16:35:49.357337 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.357315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"]
Sep 07 16:35:49.379340 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.379311    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"]
Sep 07 16:35:49.382634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.382620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.393713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.393690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"]
Sep 07 16:35:49.505118 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.505025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scncs\" (UniqueName: \"kubernetes.io/projected/dd546456-a651-4f22-8939-f1b182e6f527-kube-api-access-scncs\") pod \"payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.505118 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.505067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3123c5-96fe-432b-b90e-7b1e925250c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.505289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.505164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd546456-a651-4f22-8939-f1b182e6f527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.505289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.505245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdtlb\" (UniqueName: \"kubernetes.io/projected/fc3123c5-96fe-432b-b90e-7b1e925250c7-kube-api-access-kdtlb\") pod \"payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.525804 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.525773    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d1fab42-95c4-4eff-b873-0beba327abea" containerID="dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4" exitCode=0
Sep 07 16:35:49.525984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.525837    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"
Sep 07 16:35:49.525984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.525844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" event={"ID":"3d1fab42-95c4-4eff-b873-0beba327abea","Type":"ContainerDied","Data":"dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4"}
Sep 07 16:35:49.525984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.525917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm" event={"ID":"3d1fab42-95c4-4eff-b873-0beba327abea","Type":"ContainerDied","Data":"e38ef293d48365fc079a47fa9febe74c75010d5326286ace60a20ef2e806a3ea"}
Sep 07 16:35:49.525984 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.525939    2577 scope.go:117] "RemoveContainer" containerID="dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4"
Sep 07 16:35:49.535496 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.535480    2577 scope.go:117] "RemoveContainer" containerID="2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"
Sep 07 16:35:49.544223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.544205    2577 scope.go:117] "RemoveContainer" containerID="dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4"
Sep 07 16:35:49.544606 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:49.544580    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4\": container with ID starting with dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4 not found: ID does not exist" containerID="dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4"
Sep 07 16:35:49.544771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.544620    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4"} err="failed to get container status \"dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4\": rpc error: code = NotFound desc = could not find container \"dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4\": container with ID starting with dcdb3dd2eb22bd8861dd97ca9aadda571b4b3d4fccf77d0384c65997cd33bdb4 not found: ID does not exist"
Sep 07 16:35:49.544771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.544644    2577 scope.go:117] "RemoveContainer" containerID="2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"
Sep 07 16:35:49.545185 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:35:49.545153    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160\": container with ID starting with 2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160 not found: ID does not exist" containerID="2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"
Sep 07 16:35:49.545290 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.545194    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160"} err="failed to get container status \"2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160\": rpc error: code = NotFound desc = could not find container \"2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160\": container with ID starting with 2d43d8528d42ac8ecdb2044666b738b87f3c905f1d6b262747edd64858468160 not found: ID does not exist"
Sep 07 16:35:49.547036 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.547015    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"]
Sep 07 16:35:49.552804 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.552785    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a7d399-openshift-default-587867c9bb-kgwpm"]
Sep 07 16:35:49.567882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.567861    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1fab42-95c4-4eff-b873-0beba327abea" path="/var/lib/kubelet/pods/3d1fab42-95c4-4eff-b873-0beba327abea/volumes"
Sep 07 16:35:49.606168 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.606133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd546456-a651-4f22-8939-f1b182e6f527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.606315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.606189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdtlb\" (UniqueName: \"kubernetes.io/projected/fc3123c5-96fe-432b-b90e-7b1e925250c7-kube-api-access-kdtlb\") pod \"payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.606315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.606287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scncs\" (UniqueName: \"kubernetes.io/projected/dd546456-a651-4f22-8939-f1b182e6f527-kube-api-access-scncs\") pod \"payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.606400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.606312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3123c5-96fe-432b-b90e-7b1e925250c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.606914 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.606895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd546456-a651-4f22-8939-f1b182e6f527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.606985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.606967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3123c5-96fe-432b-b90e-7b1e925250c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.614654 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.614622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scncs\" (UniqueName: \"kubernetes.io/projected/dd546456-a651-4f22-8939-f1b182e6f527-kube-api-access-scncs\") pod \"payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.614747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.614620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdtlb\" (UniqueName: \"kubernetes.io/projected/fc3123c5-96fe-432b-b90e-7b1e925250c7-kube-api-access-kdtlb\") pod \"payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.657110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.657071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:49.694425 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.694381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:49.930242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.930219    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"]
Sep 07 16:35:49.932204 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:35:49.932175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd546456_a651_4f22_8939_f1b182e6f527.slice/crio-5a9f20c270b6b8fdffc06e3b33d915f747034da4b00a8dcc3807abe9068fa3e5 WatchSource:0}: Error finding container 5a9f20c270b6b8fdffc06e3b33d915f747034da4b00a8dcc3807abe9068fa3e5: Status 404 returned error can't find the container with id 5a9f20c270b6b8fdffc06e3b33d915f747034da4b00a8dcc3807abe9068fa3e5
Sep 07 16:35:49.982651 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:49.982625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"]
Sep 07 16:35:49.987100 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:35:49.987071    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3123c5_96fe_432b_b90e_7b1e925250c7.slice/crio-90e9bd32cdd5dbf8df1d13fe61bc1d0315e2bc6e1e9243757d02d46ea5cba2eb WatchSource:0}: Error finding container 90e9bd32cdd5dbf8df1d13fe61bc1d0315e2bc6e1e9243757d02d46ea5cba2eb: Status 404 returned error can't find the container with id 90e9bd32cdd5dbf8df1d13fe61bc1d0315e2bc6e1e9243757d02d46ea5cba2eb
Sep 07 16:35:50.534778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.534715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc" event={"ID":"fc3123c5-96fe-432b-b90e-7b1e925250c7","Type":"ContainerStarted","Data":"29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab"}
Sep 07 16:35:50.534999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.534798    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:35:50.534999 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.534822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc" event={"ID":"fc3123c5-96fe-432b-b90e-7b1e925250c7","Type":"ContainerStarted","Data":"90e9bd32cdd5dbf8df1d13fe61bc1d0315e2bc6e1e9243757d02d46ea5cba2eb"}
Sep 07 16:35:50.536530 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.536496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt" event={"ID":"dd546456-a651-4f22-8939-f1b182e6f527","Type":"ContainerStarted","Data":"f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e"}
Sep 07 16:35:50.536703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.536556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt" event={"ID":"dd546456-a651-4f22-8939-f1b182e6f527","Type":"ContainerStarted","Data":"5a9f20c270b6b8fdffc06e3b33d915f747034da4b00a8dcc3807abe9068fa3e5"}
Sep 07 16:35:50.536703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.536584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:35:50.552568 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.552514    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc" podStartSLOduration=1.552496789 podStartE2EDuration="1.552496789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:50.551377583 +0000 UTC m=+955.432914557" watchObservedRunningTime="2026-09-07 16:35:50.552496789 +0000 UTC m=+955.434033783"
Sep 07 16:35:50.567133 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.567080    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt" podStartSLOduration=1.567061869 podStartE2EDuration="1.567061869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:50.566019767 +0000 UTC m=+955.447556735" watchObservedRunningTime="2026-09-07 16:35:50.567061869 +0000 UTC m=+955.448598833"
Sep 07 16:35:50.901894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:50.901848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:51.002149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:51.002112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:51.140167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:51.140135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:51.445461 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:51.445416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:51.545390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:51.545362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:51.645625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:51.645580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:57.532686 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:57.532641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:57.642713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:57.642676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:57.732878 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:57.732843    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:35:57.855998 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:35:57.855904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:01.546497 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:01.546460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:36:01.546986 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:01.546546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:36:03.791263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:03.791232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:03.893541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:03.893494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:04.688518 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:04.688484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:04.981207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:04.981129    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"]
Sep 07 16:36:04.986546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:04.986522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:04.989305 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:04.989276    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bd6c5a-openshift-default-dockercfg-jmbz8\""
Sep 07 16:36:04.997629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:04.997592    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"]
Sep 07 16:36:05.006785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.006737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:05.061614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-token\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a54fcc3-20df-4f2e-8d63-2728436f6156-istiod-ca-cert\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6ts\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-kube-api-access-4d6ts\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-data\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-socket\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-credential-socket\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-envoy\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.061945 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-certs\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.062248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.061982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-podinfo\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.162849 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.162800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6ts\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-kube-api-access-4d6ts\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.162858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-data\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.162899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-socket\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.162914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-credential-socket\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.162958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-envoy\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163044 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-certs\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-podinfo\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-token\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a54fcc3-20df-4f2e-8d63-2728436f6156-istiod-ca-cert\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163457 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-socket\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163457 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-credential-socket\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163457 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-data\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163616 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-certs\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.163879 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.163857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a54fcc3-20df-4f2e-8d63-2728436f6156-istiod-ca-cert\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.165796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.165769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-envoy\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.166322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.166302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-podinfo\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.171335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.171310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6ts\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-kube-api-access-4d6ts\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.171452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.171414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-token\") pod \"e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") " pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.227652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.227618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:05.299075 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.298985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:05.587843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.587810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"]
Sep 07 16:36:05.590107 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:36:05.590075    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a54fcc3_20df_4f2e_8d63_2728436f6156.slice/crio-b3b03d850eafc5971e419cee2bc679057d32de10606d864977b34b960a67220a WatchSource:0}: Error finding container b3b03d850eafc5971e419cee2bc679057d32de10606d864977b34b960a67220a: Status 404 returned error can't find the container with id b3b03d850eafc5971e419cee2bc679057d32de10606d864977b34b960a67220a
Sep 07 16:36:05.592765 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.592715    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:05.592866 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.592816    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:05.592866 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.592860    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:05.621921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:05.621883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" event={"ID":"0a54fcc3-20df-4f2e-8d63-2728436f6156","Type":"ContainerStarted","Data":"b3b03d850eafc5971e419cee2bc679057d32de10606d864977b34b960a67220a"}
Sep 07 16:36:06.628708 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:06.628669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" event={"ID":"0a54fcc3-20df-4f2e-8d63-2728436f6156","Type":"ContainerStarted","Data":"d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"}
Sep 07 16:36:06.649042 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:06.648976    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podStartSLOduration=2.6489568070000002 podStartE2EDuration="2.648956807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:06.648083939 +0000 UTC m=+971.529620909" watchObservedRunningTime="2026-09-07 16:36:06.648956807 +0000 UTC m=+971.530493775"
Sep 07 16:36:07.299327 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.299284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:07.304569 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.304542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:07.635005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.634909    2577 generic.go:356] "Generic (PLEG): container finished" podID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerID="dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470" exitCode=0
Sep 07 16:36:07.635005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.634986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" event={"ID":"89bffe79-62d9-4cbf-864d-fbe1067631eb","Type":"ContainerDied","Data":"dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"}
Sep 07 16:36:07.635370 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.635223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:07.635461 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.635436    2577 scope.go:117] "RemoveContainer" containerID="dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"
Sep 07 16:36:07.636346 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.636314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:07.636492 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.636464    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:07.636549 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.636536    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:07.636586 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:07.636572    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:08.513905 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:08.513870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:08.615263 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:08.615217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:08.642693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:08.642665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" event={"ID":"89bffe79-62d9-4cbf-864d-fbe1067631eb","Type":"ContainerStarted","Data":"5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af"}
Sep 07 16:36:08.730648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:08.730600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:36:08.730923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:08.730899    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 07 16:36:08.731007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:08.730957    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 07 16:36:09.731257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:09.731223    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 07 16:36:09.731634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:09.731278    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 07 16:36:10.266891 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.266855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:10.333850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.333807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:10.667303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.667239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:10.731502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.731465    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 07 16:36:10.731998 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.731566    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 07 16:36:10.827865 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.827777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:10.920895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.920777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:10.921798 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.921747    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"]
Sep 07 16:36:10.930546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.930518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:10.936117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.935889    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bd6c5a\""
Sep 07 16:36:10.936117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.935992    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bd6c5a-dockercfg-k9szt\""
Sep 07 16:36:10.936678 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:10.936646    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"]
Sep 07 16:36:11.035074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.035028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc084ea-5dcf-401d-ba91-52d602a4ed57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.035259 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.035181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm24d\" (UniqueName: \"kubernetes.io/projected/8bc084ea-5dcf-401d-ba91-52d602a4ed57-kube-api-access-pm24d\") pod \"payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.136624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.136583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc084ea-5dcf-401d-ba91-52d602a4ed57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.136870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.136644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm24d\" (UniqueName: \"kubernetes.io/projected/8bc084ea-5dcf-401d-ba91-52d602a4ed57-kube-api-access-pm24d\") pod \"payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.137406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.137379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc084ea-5dcf-401d-ba91-52d602a4ed57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.144649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.144624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm24d\" (UniqueName: \"kubernetes.io/projected/8bc084ea-5dcf-401d-ba91-52d602a4ed57-kube-api-access-pm24d\") pod \"payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.194491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.194396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:11.244625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.244581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.523053 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.523022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"]
Sep 07 16:36:11.524901 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:36:11.524850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc084ea_5dcf_401d_ba91_52d602a4ed57.slice/crio-bcb8d7a32b528a00ead3927f26f4b9b958f00d1ec515cf1caccd63705242f7c7 WatchSource:0}: Error finding container bcb8d7a32b528a00ead3927f26f4b9b958f00d1ec515cf1caccd63705242f7c7: Status 404 returned error can't find the container with id bcb8d7a32b528a00ead3927f26f4b9b958f00d1ec515cf1caccd63705242f7c7
Sep 07 16:36:11.662089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.662050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" event={"ID":"8bc084ea-5dcf-401d-ba91-52d602a4ed57","Type":"ContainerStarted","Data":"a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e"}
Sep 07 16:36:11.662266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.662111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" event={"ID":"8bc084ea-5dcf-401d-ba91-52d602a4ed57","Type":"ContainerStarted","Data":"bcb8d7a32b528a00ead3927f26f4b9b958f00d1ec515cf1caccd63705242f7c7"}
Sep 07 16:36:11.662266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.662161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:11.680169 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.680124    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" podStartSLOduration=1.6801093059999999 podStartE2EDuration="1.680109306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:11.678363559 +0000 UTC m=+976.559900529" watchObservedRunningTime="2026-09-07 16:36:11.680109306 +0000 UTC m=+976.561646274"
Sep 07 16:36:11.731435 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.731392    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 07 16:36:11.731633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.731470    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 07 16:36:11.989642 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:11.989604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:12.252191 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:12.252106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:12.564701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:12.564613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:12.731631 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:12.731587    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 07 16:36:12.732130 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:12.731674    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 07 16:36:12.866214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:12.866130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:12.965934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:12.965899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:13.956028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:13.955981    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:36:13.956553 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:13.956458    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:36:13.958679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:13.958653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:36:15.949013 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:15.948962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:16.041185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:16.041148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:17.113830 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.112275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:17.291907 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.291839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:17.385819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.385790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:17.623314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.623238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:17.711655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.711568    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:17.738103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.738071    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"]
Sep 07 16:36:17.741140 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.741120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:17.743934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.743790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7aca0f-dockercfg-gltjp\""
Sep 07 16:36:17.743934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.743787    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7aca0f\""
Sep 07 16:36:17.750114 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.750094    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"]
Sep 07 16:36:17.801031 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.800997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb01fa9f-694f-423c-8d29-36832a892c8a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:17.801167 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.801062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnz8l\" (UniqueName: \"kubernetes.io/projected/cb01fa9f-694f-423c-8d29-36832a892c8a-kube-api-access-hnz8l\") pod \"payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:17.902554 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.902511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb01fa9f-694f-423c-8d29-36832a892c8a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:17.902740 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.902597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnz8l\" (UniqueName: \"kubernetes.io/projected/cb01fa9f-694f-423c-8d29-36832a892c8a-kube-api-access-hnz8l\") pod \"payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:17.903235 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.903214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb01fa9f-694f-423c-8d29-36832a892c8a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:17.911605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:17.911552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnz8l\" (UniqueName: \"kubernetes.io/projected/cb01fa9f-694f-423c-8d29-36832a892c8a-kube-api-access-hnz8l\") pod \"payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:18.053021 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:18.052942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:18.331773 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:18.328492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"]
Sep 07 16:36:18.331773 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:36:18.331437    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb01fa9f_694f_423c_8d29_36832a892c8a.slice/crio-3aa9b3b58214791077c052042e7f6aeaa9565c2cea2c77576741ecb0d1ebcc09 WatchSource:0}: Error finding container 3aa9b3b58214791077c052042e7f6aeaa9565c2cea2c77576741ecb0d1ebcc09: Status 404 returned error can't find the container with id 3aa9b3b58214791077c052042e7f6aeaa9565c2cea2c77576741ecb0d1ebcc09
Sep 07 16:36:18.704547 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:18.704510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj" event={"ID":"cb01fa9f-694f-423c-8d29-36832a892c8a","Type":"ContainerStarted","Data":"36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85"}
Sep 07 16:36:18.704735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:18.704577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj" event={"ID":"cb01fa9f-694f-423c-8d29-36832a892c8a","Type":"ContainerStarted","Data":"3aa9b3b58214791077c052042e7f6aeaa9565c2cea2c77576741ecb0d1ebcc09"}
Sep 07 16:36:18.704735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:18.704606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:18.724190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:18.724132    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj" podStartSLOduration=1.724110961 podStartE2EDuration="1.724110961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:18.720685223 +0000 UTC m=+983.602222192" watchObservedRunningTime="2026-09-07 16:36:18.724110961 +0000 UTC m=+983.605647930"
Sep 07 16:36:22.671530 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:22.671502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:36:23.677964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:23.677916    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 16:36:23.679336 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:23.677995    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 16:36:24.744669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:24.744638    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerID="d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26" exitCode=0
Sep 07 16:36:24.745143 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:24.744716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" event={"ID":"0a54fcc3-20df-4f2e-8d63-2728436f6156","Type":"ContainerDied","Data":"d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"}
Sep 07 16:36:24.745279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:24.745262    2577 scope.go:117] "RemoveContainer" containerID="d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"
Sep 07 16:36:24.746381 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:24.746343    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:24.746485 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:24.746445    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:24.746546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:24.746499    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:25.751913 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:25.751882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" event={"ID":"0a54fcc3-20df-4f2e-8d63-2728436f6156","Type":"ContainerStarted","Data":"f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc"}
Sep 07 16:36:26.300217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:26.300177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:26.300481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:26.300456    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 07 16:36:26.300538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:26.300511    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 07 16:36:27.299818 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:27.299784    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 07 16:36:27.300193 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:27.299838    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 07 16:36:28.300119 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:28.300084    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 07 16:36:28.300555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:28.300137    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 07 16:36:29.299850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:29.299813    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 07 16:36:29.300033 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:29.299869    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 07 16:36:29.717018 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:29.716976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:36:30.299461 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:30.299426    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 07 16:36:30.299616 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:30.299478    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 07 16:36:31.219589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:31.219556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:31.303279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:31.303242    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:31.303628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:31.303611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:31.304570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:31.304545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:36:33.463178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:33.463144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:33.821627 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:33.821539    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:34.037363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.037331    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"]
Sep 07 16:36:34.040445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.040411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.042820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.042801    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4a9a21-openshift-default-dockercfg-55nvt\""
Sep 07 16:36:34.052502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.052482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"]
Sep 07 16:36:34.063805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.063780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:34.082214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-token\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082214 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5975e495-336c-4993-8bf7-ae7ed658ff4a-istiod-ca-cert\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082382 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8xn\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-kube-api-access-mg8xn\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082382 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-envoy\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-podinfo\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-data\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082493 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-certs\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082639 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-credential-socket\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.082639 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.082556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-socket\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-token\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5975e495-336c-4993-8bf7-ae7ed658ff4a-istiod-ca-cert\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8xn\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-kube-api-access-mg8xn\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-envoy\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-podinfo\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-data\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-certs\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-credential-socket\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.183912 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.183863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-socket\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.184110 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.184088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-data\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.184224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.184201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-certs\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.184224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.184215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-credential-socket\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.184357 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.184258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-socket\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.184357 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.184287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5975e495-336c-4993-8bf7-ae7ed658ff4a-istiod-ca-cert\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.186033 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.186015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-envoy\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.186278 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.186260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-podinfo\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.191828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.191810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-token\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.192007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.191992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8xn\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-kube-api-access-mg8xn\") pod \"e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") " pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.353275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.353187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:34.498552 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.498516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:34.628464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.628442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"]
Sep 07 16:36:34.630474 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:36:34.630441    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5975e495_336c_4993_8bf7_ae7ed658ff4a.slice/crio-d45911f0c1da70be2adcae8eff00a0841efb41b9a45f7a6ded1d771d4e5107bf WatchSource:0}: Error finding container d45911f0c1da70be2adcae8eff00a0841efb41b9a45f7a6ded1d771d4e5107bf: Status 404 returned error can't find the container with id d45911f0c1da70be2adcae8eff00a0841efb41b9a45f7a6ded1d771d4e5107bf
Sep 07 16:36:34.633163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.633131    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:34.633269 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.633209    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:34.633269 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.633258    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:34.808531 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.808495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" event={"ID":"5975e495-336c-4993-8bf7-ae7ed658ff4a","Type":"ContainerStarted","Data":"abe47834c7ce96344cc321a8d8476075323c69c07716fe02abe885e4d2a732da"}
Sep 07 16:36:34.808732 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.808552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" event={"ID":"5975e495-336c-4993-8bf7-ae7ed658ff4a","Type":"ContainerStarted","Data":"d45911f0c1da70be2adcae8eff00a0841efb41b9a45f7a6ded1d771d4e5107bf"}
Sep 07 16:36:34.830942 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:34.830884    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podStartSLOduration=0.830857207 podStartE2EDuration="830.857207ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:34.825921806 +0000 UTC m=+999.707458809" watchObservedRunningTime="2026-09-07 16:36:34.830857207 +0000 UTC m=+999.712394177"
Sep 07 16:36:35.353668 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:35.353634    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:35.358799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:35.358779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:35.814428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:35.814396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:35.815332 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:35.815311    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:36.717971 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:36.717939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:36.806577 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:36.806548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:37.962580 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:37.962549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:38.063617 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.063585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:38.170443 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.170409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:38.406596 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.406571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:38.503406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.503380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:38.512262 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.512239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"]
Sep 07 16:36:38.514795 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.514781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.517258 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.517238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4a9a21\""
Sep 07 16:36:38.517434 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.517415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4a9a21-dockercfg-lrnnf\""
Sep 07 16:36:38.526718 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.526692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"]
Sep 07 16:36:38.624950 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.624915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85408405-b3b1-4cf7-aa34-a19ee068886e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.625139 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.625072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxc4\" (UniqueName: \"kubernetes.io/projected/85408405-b3b1-4cf7-aa34-a19ee068886e-kube-api-access-cwxc4\") pod \"payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.726821 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.726687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxc4\" (UniqueName: \"kubernetes.io/projected/85408405-b3b1-4cf7-aa34-a19ee068886e-kube-api-access-cwxc4\") pod \"payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.726976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.726832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85408405-b3b1-4cf7-aa34-a19ee068886e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.727538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.727518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85408405-b3b1-4cf7-aa34-a19ee068886e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.734388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.734365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxc4\" (UniqueName: \"kubernetes.io/projected/85408405-b3b1-4cf7-aa34-a19ee068886e-kube-api-access-cwxc4\") pod \"payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:38.825731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:38.825696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:39.096772 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.096733    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"]
Sep 07 16:36:39.098440 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:36:39.098413    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85408405_b3b1_4cf7_aa34_a19ee068886e.slice/crio-62c235b601d40e6dd57ed0dd034234d983e19805883c4736c010fb425a099d4a WatchSource:0}: Error finding container 62c235b601d40e6dd57ed0dd034234d983e19805883c4736c010fb425a099d4a: Status 404 returned error can't find the container with id 62c235b601d40e6dd57ed0dd034234d983e19805883c4736c010fb425a099d4a
Sep 07 16:36:39.166950 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.166912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:39.837622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.837592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd" event={"ID":"85408405-b3b1-4cf7-aa34-a19ee068886e","Type":"ContainerStarted","Data":"24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e"}
Sep 07 16:36:39.837830 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.837647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd" event={"ID":"85408405-b3b1-4cf7-aa34-a19ee068886e","Type":"ContainerStarted","Data":"62c235b601d40e6dd57ed0dd034234d983e19805883c4736c010fb425a099d4a"}
Sep 07 16:36:39.837830 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.837721    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:39.854792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.854716    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd" podStartSLOduration=1.8546973580000001 podStartE2EDuration="1.854697358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:39.852557403 +0000 UTC m=+1004.734094373" watchObservedRunningTime="2026-09-07 16:36:39.854697358 +0000 UTC m=+1004.736234328"
Sep 07 16:36:39.971186 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:39.971140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:40.607919 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:40.607888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:40.860386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:40.860270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:48.833563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.833529    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"]
Sep 07 16:36:48.839935 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.839908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:48.842734 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.842713    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"kube-root-ca.crt\""
Sep 07 16:36:48.843005 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.842857    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"openshift-service-ca.crt\""
Sep 07 16:36:48.843886 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.843864    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"default-dockercfg-qcns6\""
Sep 07 16:36:48.844145 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.844127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"auth-test-model-8d59e2-kserve-self-signed-certs\""
Sep 07 16:36:48.846123 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.846073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"]
Sep 07 16:36:48.865396 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.865367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:48.933400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.933372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-dshm\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:48.933593 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.933516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-model-cache\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:48.933667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.933601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhd6l\" (UniqueName: \"kubernetes.io/projected/2a9c2fd8-8097-43ec-9a8c-27d811549250-kube-api-access-hhd6l\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:48.933667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.933642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-tmp-dir\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:48.933792 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.933717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-home\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:48.933859 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:48.933797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9c2fd8-8097-43ec-9a8c-27d811549250-tls-certs\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.034977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.034925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9c2fd8-8097-43ec-9a8c-27d811549250-tls-certs\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.034977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.034982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-dshm\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-model-cache\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhd6l\" (UniqueName: \"kubernetes.io/projected/2a9c2fd8-8097-43ec-9a8c-27d811549250-kube-api-access-hhd6l\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-tmp-dir\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-home\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035548 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-model-cache\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035548 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-tmp-dir\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.035690 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.035586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-home\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.037699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.037675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-dshm\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.037935 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.037914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9c2fd8-8097-43ec-9a8c-27d811549250-tls-certs\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.043123 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.043076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhd6l\" (UniqueName: \"kubernetes.io/projected/2a9c2fd8-8097-43ec-9a8c-27d811549250-kube-api-access-hhd6l\") pod \"auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.153236 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.153187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:49.431222 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.431188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"]
Sep 07 16:36:49.434184 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:36:49.434147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a9c2fd8_8097_43ec_9a8c_27d811549250.slice/crio-cad79ace9c9557cc70f03e08cfe903f774e49b8eafe9a43389628053acd78c0d WatchSource:0}: Error finding container cad79ace9c9557cc70f03e08cfe903f774e49b8eafe9a43389628053acd78c0d: Status 404 returned error can't find the container with id cad79ace9c9557cc70f03e08cfe903f774e49b8eafe9a43389628053acd78c0d
Sep 07 16:36:49.894077 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.894025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8" event={"ID":"2a9c2fd8-8097-43ec-9a8c-27d811549250","Type":"ContainerStarted","Data":"cd1c528ef34d942bad384c8032731db28aa90e259134ba1ba8806ee053ce85cd"}
Sep 07 16:36:49.894468 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.894085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8" event={"ID":"2a9c2fd8-8097-43ec-9a8c-27d811549250","Type":"ContainerStarted","Data":"cad79ace9c9557cc70f03e08cfe903f774e49b8eafe9a43389628053acd78c0d"}
Sep 07 16:36:49.910148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:49.910097    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8" podStartSLOduration=1.910083136 podStartE2EDuration="1.910083136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:49.909232461 +0000 UTC m=+1014.790769441" watchObservedRunningTime="2026-09-07 16:36:49.910083136 +0000 UTC m=+1014.791620105"
Sep 07 16:36:50.846578 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:50.846543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:36:51.855988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:51.855948    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 16:36:51.856514 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:51.856020    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 16:36:52.912923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:52.912825    2577 generic.go:356] "Generic (PLEG): container finished" podID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerID="abe47834c7ce96344cc321a8d8476075323c69c07716fe02abe885e4d2a732da" exitCode=0
Sep 07 16:36:52.912923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:52.912899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" event={"ID":"5975e495-336c-4993-8bf7-ae7ed658ff4a","Type":"ContainerDied","Data":"abe47834c7ce96344cc321a8d8476075323c69c07716fe02abe885e4d2a732da"}
Sep 07 16:36:52.913428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:52.913386    2577 scope.go:117] "RemoveContainer" containerID="abe47834c7ce96344cc321a8d8476075323c69c07716fe02abe885e4d2a732da"
Sep 07 16:36:52.914675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:52.914646    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:52.914810 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:52.914730    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:52.914885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:52.914811    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:36:53.926345 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:53.926301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" event={"ID":"5975e495-336c-4993-8bf7-ae7ed658ff4a","Type":"ContainerStarted","Data":"88617b06bc3575775a63df46e950932c6863da013973c2d8d939f05e53756573"}
Sep 07 16:36:54.353916 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:54.353809    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:36:54.354131 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:54.354108    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 16:36:54.354212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:54.354163    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 16:36:55.353834 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:55.353793    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 16:36:55.354328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:55.353861    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 16:36:56.354662 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:56.354621    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 16:36:56.355115 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:56.354677    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 16:36:57.354122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:57.354082    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 16:36:57.354308 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:57.354142    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 16:36:58.353684 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:58.353644    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 16:36:58.354178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:58.353702    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 16:36:58.855869 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:58.855835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:36:59.154141 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:59.154105    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:59.154394 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:59.154376    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:59.167349 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:59.167321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:36:59.354329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:59.354285    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 16:36:59.354697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:59.354350    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 16:36:59.975820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:36:59.975790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:37:00.357679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:00.357597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:37:00.358103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:00.358020    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:37:00.358745 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:00.358729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:37:01.619723 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:01.619688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:09.830937 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.830896    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"]
Sep 07 16:37:09.834662 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.834637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.837427 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.837406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"openshift-service-ca.crt\""
Sep 07 16:37:09.838479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.838420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"kube-root-ca.crt\""
Sep 07 16:37:09.838479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.838464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"auth-test-model-7aca0f-kserve-self-signed-certs\""
Sep 07 16:37:09.838652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.838527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"default-dockercfg-d4jx5\""
Sep 07 16:37:09.843694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.843672    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"]
Sep 07 16:37:09.851787 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.851742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tls-certs\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.851889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.851799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-dshm\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.851889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.851836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-home\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.851889 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.851858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tmp-dir\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.851991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.851965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4745\" (UniqueName: \"kubernetes.io/projected/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-kube-api-access-j4745\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.852027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.851997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-model-cache\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.855435 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.855416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:09.953198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-model-cache\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tls-certs\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-dshm\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-home\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953383 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tmp-dir\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953571 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4745\" (UniqueName: \"kubernetes.io/projected/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-kube-api-access-j4745\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953680 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-model-cache\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953779 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-home\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.953847 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.953809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tmp-dir\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.955670 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.955648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-dshm\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.955874 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.955857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tls-certs\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:09.961334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:09.961312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4745\" (UniqueName: \"kubernetes.io/projected/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-kube-api-access-j4745\") pod \"auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:10.145551 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:10.145525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:10.420277 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:10.420246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"]
Sep 07 16:37:10.421928 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:37:10.421901    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a3bef08_d41e_4a1d_8d0e_84a23c8f7cfc.slice/crio-e35de632a42bf835b4fd8f259c9884d93134f0a484e4650ce9698364bf2b2b5a WatchSource:0}: Error finding container e35de632a42bf835b4fd8f259c9884d93134f0a484e4650ce9698364bf2b2b5a: Status 404 returned error can't find the container with id e35de632a42bf835b4fd8f259c9884d93134f0a484e4650ce9698364bf2b2b5a
Sep 07 16:37:11.025080 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:11.025041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn" event={"ID":"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc","Type":"ContainerStarted","Data":"5a2ae698a362cb658aa91f653682ee9d07a6f52944a587028e61e2f920854a71"}
Sep 07 16:37:11.025467 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:11.025091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn" event={"ID":"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc","Type":"ContainerStarted","Data":"e35de632a42bf835b4fd8f259c9884d93134f0a484e4650ce9698364bf2b2b5a"}
Sep 07 16:37:11.044563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:11.044514    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn" podStartSLOduration=2.044498321 podStartE2EDuration="2.044498321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:37:11.042693389 +0000 UTC m=+1035.924230362" watchObservedRunningTime="2026-09-07 16:37:11.044498321 +0000 UTC m=+1035.926035328"
Sep 07 16:37:15.155148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:15.155116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:15.455248 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:15.455160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:15.771552 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:15.771461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:18.136429 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:18.136392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:18.229323 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:18.229287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:19.383060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:19.383012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:19.484145 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:19.484112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:19.584098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:19.584052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:19.836763 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:19.836716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:19.937646 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:19.937614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:20.146426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:20.146385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:20.146629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:20.146448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:20.158872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:20.158845    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:21.097613 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:21.097575    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:37:23.963170 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:23.963135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:41.680870 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:41.680827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:41.862512 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:41.862483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:43.035178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:43.035148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:50.233840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.233808    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"]
Sep 07 16:37:50.238875 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.238843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.241391 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.241360    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2ff801c7\"/\"openshift-service-ca.crt\""
Sep 07 16:37:50.242578 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.242553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2ff801c7\"/\"kube-root-ca.crt\""
Sep 07 16:37:50.242895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.242873    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2ff801c7\"/\"e2e-lifecycle-model-2ff801c7-kserve-self-signed-certs\""
Sep 07 16:37:50.243028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.242946    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2ff801c7\"/\"default-dockercfg-m966w\""
Sep 07 16:37:50.245733 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.245706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"]
Sep 07 16:37:50.253483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.253455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:37:50.264390 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.264360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-home\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.264555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.264424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tls-certs\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.264555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.264446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tmp-dir\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.264555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.264483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8d49\" (UniqueName: \"kubernetes.io/projected/7b837a3d-d1b2-4801-bb18-f451c8ed3997-kube-api-access-d8d49\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.264555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.264522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-dshm\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.264710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.264560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-model-cache\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.365561 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.365521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d49\" (UniqueName: \"kubernetes.io/projected/7b837a3d-d1b2-4801-bb18-f451c8ed3997-kube-api-access-d8d49\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.365805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.365586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-dshm\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.365805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.365644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-model-cache\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.365805 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.365714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-home\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.365980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.365809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tls-certs\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.365980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.365834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tmp-dir\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.366191 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.366161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-home\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.366311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.366171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-model-cache\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.366311 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.366267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tmp-dir\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.368049 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.368027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-dshm\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.368352 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.368327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tls-certs\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.374160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.374130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d49\" (UniqueName: \"kubernetes.io/projected/7b837a3d-d1b2-4801-bb18-f451c8ed3997-kube-api-access-d8d49\") pod \"e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") " pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.550480 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.550396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:37:50.824264 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:50.824240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"]
Sep 07 16:37:50.826266 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:37:50.826236    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b837a3d_d1b2_4801_bb18_f451c8ed3997.slice/crio-9c8e0387cf3cf20882fa3745387740896b15a2d0f8c4742c5ac7557ae8b96736 WatchSource:0}: Error finding container 9c8e0387cf3cf20882fa3745387740896b15a2d0f8c4742c5ac7557ae8b96736: Status 404 returned error can't find the container with id 9c8e0387cf3cf20882fa3745387740896b15a2d0f8c4742c5ac7557ae8b96736
Sep 07 16:37:51.248807 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:51.248771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb" event={"ID":"7b837a3d-d1b2-4801-bb18-f451c8ed3997","Type":"ContainerStarted","Data":"a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb"}
Sep 07 16:37:51.249190 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:51.248813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb" event={"ID":"7b837a3d-d1b2-4801-bb18-f451c8ed3997","Type":"ContainerStarted","Data":"9c8e0387cf3cf20882fa3745387740896b15a2d0f8c4742c5ac7557ae8b96736"}
Sep 07 16:37:51.265058 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:37:51.265006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb" podStartSLOduration=1.264991266 podStartE2EDuration="1.264991266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:37:51.263936538 +0000 UTC m=+1076.145473519" watchObservedRunningTime="2026-09-07 16:37:51.264991266 +0000 UTC m=+1076.146528279"
Sep 07 16:38:00.550634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:00.550586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:38:00.551100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:00.550651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:38:00.563017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:00.562990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:38:01.316244 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:01.316210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:38:16.241721 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:16.241685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:17.174358 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:17.174327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:32.497670 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:32.497636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:32.562298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:32.562265    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:32.633968 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:32.633940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:32.733180 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:32.733149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:35.621183 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:35.621150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:40.605412 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:40.605373    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"]
Sep 07 16:38:40.647084 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:40.647038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:38:55.625462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:55.625421    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb" podUID="7b837a3d-d1b2-4801-bb18-f451c8ed3997" containerName="main" containerID="cri-o://a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb" gracePeriod=45
Sep 07 16:38:55.892591 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:55.892566    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:38:56.026127 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-dshm\") pod \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") "
Sep 07 16:38:56.026322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-home\") pod \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") "
Sep 07 16:38:56.026322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8d49\" (UniqueName: \"kubernetes.io/projected/7b837a3d-d1b2-4801-bb18-f451c8ed3997-kube-api-access-d8d49\") pod \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") "
Sep 07 16:38:56.026322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026233    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tls-certs\") pod \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") "
Sep 07 16:38:56.026322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tmp-dir\") pod \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") "
Sep 07 16:38:56.026534 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-model-cache\") pod \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\" (UID: \"7b837a3d-d1b2-4801-bb18-f451c8ed3997\") "
Sep 07 16:38:56.026534 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-home" (OuterVolumeSpecName: "home") pod "7b837a3d-d1b2-4801-bb18-f451c8ed3997" (UID: "7b837a3d-d1b2-4801-bb18-f451c8ed3997"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:38:56.026739 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026721    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:38:56.026940 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026747    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b837a3d-d1b2-4801-bb18-f451c8ed3997" (UID: "7b837a3d-d1b2-4801-bb18-f451c8ed3997"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:38:56.026940 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.026785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b837a3d-d1b2-4801-bb18-f451c8ed3997" (UID: "7b837a3d-d1b2-4801-bb18-f451c8ed3997"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:38:56.028644 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.028612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b837a3d-d1b2-4801-bb18-f451c8ed3997" (UID: "7b837a3d-d1b2-4801-bb18-f451c8ed3997"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:38:56.028741 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.028645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-dshm" (OuterVolumeSpecName: "dshm") pod "7b837a3d-d1b2-4801-bb18-f451c8ed3997" (UID: "7b837a3d-d1b2-4801-bb18-f451c8ed3997"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:38:56.028741 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.028694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b837a3d-d1b2-4801-bb18-f451c8ed3997-kube-api-access-d8d49" (OuterVolumeSpecName: "kube-api-access-d8d49") pod "7b837a3d-d1b2-4801-bb18-f451c8ed3997" (UID: "7b837a3d-d1b2-4801-bb18-f451c8ed3997"). InnerVolumeSpecName "kube-api-access-d8d49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:38:56.127709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.127675    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:38:56.127709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.127702    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:38:56.127709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.127712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8d49\" (UniqueName: \"kubernetes.io/projected/7b837a3d-d1b2-4801-bb18-f451c8ed3997-kube-api-access-d8d49\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:38:56.127985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.127721    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:38:56.127985 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.127786    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b837a3d-d1b2-4801-bb18-f451c8ed3997-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:38:56.625153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.625119    2577 generic.go:356] "Generic (PLEG): container finished" podID="7b837a3d-d1b2-4801-bb18-f451c8ed3997" containerID="a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb" exitCode=0
Sep 07 16:38:56.625324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.625163    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"
Sep 07 16:38:56.625324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.625203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb" event={"ID":"7b837a3d-d1b2-4801-bb18-f451c8ed3997","Type":"ContainerDied","Data":"a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb"}
Sep 07 16:38:56.625324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.625241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb" event={"ID":"7b837a3d-d1b2-4801-bb18-f451c8ed3997","Type":"ContainerDied","Data":"9c8e0387cf3cf20882fa3745387740896b15a2d0f8c4742c5ac7557ae8b96736"}
Sep 07 16:38:56.625324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.625259    2577 scope.go:117] "RemoveContainer" containerID="a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb"
Sep 07 16:38:56.638428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.638402    2577 scope.go:117] "RemoveContainer" containerID="a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb"
Sep 07 16:38:56.638724 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:38:56.638703    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb\": container with ID starting with a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb not found: ID does not exist" containerID="a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb"
Sep 07 16:38:56.638806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.638731    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb"} err="failed to get container status \"a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb\": rpc error: code = NotFound desc = could not find container \"a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb\": container with ID starting with a29255d44e2ec1a6c78ad3023404c41e76fe5a18bcbb9a3df2c8763ddf8249bb not found: ID does not exist"
Sep 07 16:38:56.647439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.647417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"]
Sep 07 16:38:56.650389 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:56.650366    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2ff801c7/e2e-lifecycle-model-2ff801c7-kserve-7db6df8ff9-fldgb"]
Sep 07 16:38:57.568138 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:38:57.568105    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b837a3d-d1b2-4801-bb18-f451c8ed3997" path="/var/lib/kubelet/pods/7b837a3d-d1b2-4801-bb18-f451c8ed3997/volumes"
Sep 07 16:39:05.511151 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:05.511114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:12.614471 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:12.614423    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"]
Sep 07 16:39:12.634378 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:12.634345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:13.645097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:13.645060    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"]
Sep 07 16:39:13.663820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:13.663785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:16.798564 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:16.798453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:17.844462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:17.844430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:18.372235 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:18.372178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:18.418946 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:18.418919    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:20.755149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:20.755102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:23.383151 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:23.383125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:23.483472 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:23.483438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:23.623366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:23.623338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:24.844520 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:24.844487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:24.944368 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:24.944337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:25.042905 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:25.042867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:25.607154 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:25.607123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:25.701677 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:25.701641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:25.803802 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:25.803747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:26.937319 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:26.937286    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:27.635324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.635267    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8" podUID="2a9c2fd8-8097-43ec-9a8c-27d811549250" containerName="main" containerID="cri-o://cd1c528ef34d942bad384c8032731db28aa90e259134ba1ba8806ee053ce85cd" gracePeriod=45
Sep 07 16:39:27.799498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.799466    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a9c2fd8-8097-43ec-9a8c-27d811549250" containerID="cd1c528ef34d942bad384c8032731db28aa90e259134ba1ba8806ee053ce85cd" exitCode=0
Sep 07 16:39:27.799669 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.799546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8" event={"ID":"2a9c2fd8-8097-43ec-9a8c-27d811549250","Type":"ContainerDied","Data":"cd1c528ef34d942bad384c8032731db28aa90e259134ba1ba8806ee053ce85cd"}
Sep 07 16:39:27.891896 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.891828    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:39:27.963968 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.963939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-dshm\") pod \"2a9c2fd8-8097-43ec-9a8c-27d811549250\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") "
Sep 07 16:39:27.963968 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.963970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-tmp-dir\") pod \"2a9c2fd8-8097-43ec-9a8c-27d811549250\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") "
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.963994    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-model-cache\") pod \"2a9c2fd8-8097-43ec-9a8c-27d811549250\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") "
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964058    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhd6l\" (UniqueName: \"kubernetes.io/projected/2a9c2fd8-8097-43ec-9a8c-27d811549250-kube-api-access-hhd6l\") pod \"2a9c2fd8-8097-43ec-9a8c-27d811549250\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") "
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-home\") pod \"2a9c2fd8-8097-43ec-9a8c-27d811549250\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") "
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964181    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9c2fd8-8097-43ec-9a8c-27d811549250-tls-certs\") pod \"2a9c2fd8-8097-43ec-9a8c-27d811549250\" (UID: \"2a9c2fd8-8097-43ec-9a8c-27d811549250\") "
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a9c2fd8-8097-43ec-9a8c-27d811549250" (UID: "2a9c2fd8-8097-43ec-9a8c-27d811549250"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a9c2fd8-8097-43ec-9a8c-27d811549250" (UID: "2a9c2fd8-8097-43ec-9a8c-27d811549250"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:27.964440 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-home" (OuterVolumeSpecName: "home") pod "2a9c2fd8-8097-43ec-9a8c-27d811549250" (UID: "2a9c2fd8-8097-43ec-9a8c-27d811549250"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:27.964783 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964618    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:27.964783 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964637    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:27.964783 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.964651    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:27.966397 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.966372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9c2fd8-8097-43ec-9a8c-27d811549250-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a9c2fd8-8097-43ec-9a8c-27d811549250" (UID: "2a9c2fd8-8097-43ec-9a8c-27d811549250"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:39:27.966503 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.966417    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-dshm" (OuterVolumeSpecName: "dshm") pod "2a9c2fd8-8097-43ec-9a8c-27d811549250" (UID: "2a9c2fd8-8097-43ec-9a8c-27d811549250"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:27.966503 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:27.966432    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9c2fd8-8097-43ec-9a8c-27d811549250-kube-api-access-hhd6l" (OuterVolumeSpecName: "kube-api-access-hhd6l") pod "2a9c2fd8-8097-43ec-9a8c-27d811549250" (UID: "2a9c2fd8-8097-43ec-9a8c-27d811549250"). InnerVolumeSpecName "kube-api-access-hhd6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:39:28.065452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.065418    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9c2fd8-8097-43ec-9a8c-27d811549250-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:28.065452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.065447    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhd6l\" (UniqueName: \"kubernetes.io/projected/2a9c2fd8-8097-43ec-9a8c-27d811549250-kube-api-access-hhd6l\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:28.065452 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.065458    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9c2fd8-8097-43ec-9a8c-27d811549250-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:28.665835 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.665797    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn" podUID="4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" containerName="main" containerID="cri-o://5a2ae698a362cb658aa91f653682ee9d07a6f52944a587028e61e2f920854a71" gracePeriod=45
Sep 07 16:39:28.811345 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.811315    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" containerID="5a2ae698a362cb658aa91f653682ee9d07a6f52944a587028e61e2f920854a71" exitCode=0
Sep 07 16:39:28.811533 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.811410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn" event={"ID":"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc","Type":"ContainerDied","Data":"5a2ae698a362cb658aa91f653682ee9d07a6f52944a587028e61e2f920854a71"}
Sep 07 16:39:28.813331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.813305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8" event={"ID":"2a9c2fd8-8097-43ec-9a8c-27d811549250","Type":"ContainerDied","Data":"cad79ace9c9557cc70f03e08cfe903f774e49b8eafe9a43389628053acd78c0d"}
Sep 07 16:39:28.813502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.813351    2577 scope.go:117] "RemoveContainer" containerID="cd1c528ef34d942bad384c8032731db28aa90e259134ba1ba8806ee053ce85cd"
Sep 07 16:39:28.813502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.813405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"
Sep 07 16:39:28.839718 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.839686    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"]
Sep 07 16:39:28.843568 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.843541    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/auth-test-model-8d59e2-kserve-648bf78ddc-dtdj8"]
Sep 07 16:39:28.933116 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:28.933041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:39:29.077972 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.077933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-home\") pod \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") "
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.077987    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4745\" (UniqueName: \"kubernetes.io/projected/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-kube-api-access-j4745\") pod \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") "
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tls-certs\") pod \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") "
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-model-cache\") pod \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") "
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tmp-dir\") pod \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") "
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-home" (OuterVolumeSpecName: "home") pod "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" (UID: "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-dshm\") pod \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\" (UID: \"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc\") "
Sep 07 16:39:29.078423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" (UID: "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:29.078784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" (UID: "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:29.078784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078562    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:29.078784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078579    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:29.078784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.078591    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:29.080442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.080423    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" (UID: "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:39:29.080517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.080434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-dshm" (OuterVolumeSpecName: "dshm") pod "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" (UID: "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:39:29.080517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.080426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-kube-api-access-j4745" (OuterVolumeSpecName: "kube-api-access-j4745") pod "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" (UID: "4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc"). InnerVolumeSpecName "kube-api-access-j4745". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:39:29.179947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.179901    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:29.179947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.179941    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4745\" (UniqueName: \"kubernetes.io/projected/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-kube-api-access-j4745\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:29.179947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.179952    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:39:29.568348 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.568316    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a9c2fd8-8097-43ec-9a8c-27d811549250" path="/var/lib/kubelet/pods/2a9c2fd8-8097-43ec-9a8c-27d811549250/volumes"
Sep 07 16:39:29.820048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.819960    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"
Sep 07 16:39:29.820048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.819979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn" event={"ID":"4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc","Type":"ContainerDied","Data":"e35de632a42bf835b4fd8f259c9884d93134f0a484e4650ce9698364bf2b2b5a"}
Sep 07 16:39:29.820048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.820026    2577 scope.go:117] "RemoveContainer" containerID="5a2ae698a362cb658aa91f653682ee9d07a6f52944a587028e61e2f920854a71"
Sep 07 16:39:29.838179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.838155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"]
Sep 07 16:39:29.842016 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:29.841993    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/auth-test-model-7aca0f-kserve-686dfbf96c-nmrgn"]
Sep 07 16:39:31.569339 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:31.569303    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" path="/var/lib/kubelet/pods/4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc/volumes"
Sep 07 16:39:32.705614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:32.705579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:32.806298 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:32.806262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:33.346843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:33.346815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:39.164916 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:39.164876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:39.267423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:39.267383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:39.365630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:39.365600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:44.025102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025068    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"]
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025482    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9c2fd8-8097-43ec-9a8c-27d811549250" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025494    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9c2fd8-8097-43ec-9a8c-27d811549250" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025516    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025522    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025532    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b837a3d-d1b2-4801-bb18-f451c8ed3997" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025538    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b837a3d-d1b2-4801-bb18-f451c8ed3997" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025600    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b837a3d-d1b2-4801-bb18-f451c8ed3997" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025610    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3bef08-d41e-4a1d-8d0e-84a23c8f7cfc" containerName="main"
Sep 07 16:39:44.025666 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.025615    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9c2fd8-8097-43ec-9a8c-27d811549250" containerName="main"
Sep 07 16:39:44.029764 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.029727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.034427 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.034407    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"sub-test-model-7aca0f-kserve-self-signed-certs\""
Sep 07 16:39:44.034543 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.034527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"openshift-service-ca.crt\""
Sep 07 16:39:44.034642 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.034608    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"kube-root-ca.crt\""
Sep 07 16:39:44.034789 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.034725    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"default-dockercfg-d4jx5\""
Sep 07 16:39:44.039359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.039334    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"]
Sep 07 16:39:44.059814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.059723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:44.131096 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.131056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-dshm\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.131096 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.131101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tmp-dir\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.131331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.131206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpxs\" (UniqueName: \"kubernetes.io/projected/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-kube-api-access-btpxs\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.131331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.131254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-model-cache\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.131331 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.131292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-home\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.131448 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.131327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tls-certs\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.232584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.232544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-dshm\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.232815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.232589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tmp-dir\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.232815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.232640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpxs\" (UniqueName: \"kubernetes.io/projected/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-kube-api-access-btpxs\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.232815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.232683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-model-cache\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.232815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.232722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-home\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.232815 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.232748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tls-certs\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.233148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.233109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-model-cache\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.233226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.233154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tmp-dir\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.233226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.233204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-home\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.235017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.234998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-dshm\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.235252 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.235232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tls-certs\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.240652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.240630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpxs\" (UniqueName: \"kubernetes.io/projected/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-kube-api-access-btpxs\") pod \"sub-test-model-7aca0f-kserve-54967bdc47-26dzr\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.341295 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.341191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:44.613673 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.613593    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"]
Sep 07 16:39:44.616834 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:39:44.616803    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cfc5530_13da_4ca1_8ecb_7b7d1f25d451.slice/crio-943fb696e4e6886d95539262cc00212da4be0d16f56b51d280364707e05ebb9c WatchSource:0}: Error finding container 943fb696e4e6886d95539262cc00212da4be0d16f56b51d280364707e05ebb9c: Status 404 returned error can't find the container with id 943fb696e4e6886d95539262cc00212da4be0d16f56b51d280364707e05ebb9c
Sep 07 16:39:44.903710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.903679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr" event={"ID":"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451","Type":"ContainerStarted","Data":"dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a"}
Sep 07 16:39:44.903917 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.903719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr" event={"ID":"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451","Type":"ContainerStarted","Data":"943fb696e4e6886d95539262cc00212da4be0d16f56b51d280364707e05ebb9c"}
Sep 07 16:39:44.924355 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:44.924295    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr" podStartSLOduration=0.924276557 podStartE2EDuration="924.276557ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:39:44.922015109 +0000 UTC m=+1189.803552079" watchObservedRunningTime="2026-09-07 16:39:44.924276557 +0000 UTC m=+1189.805813525"
Sep 07 16:39:49.821997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:49.821940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:50.913224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:50.913187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:39:54.342079 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:54.342042    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:54.342484 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:54.342123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:54.356344 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:54.356318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:39:54.972351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:39:54.972322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:40:10.915924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:10.915887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:16.672297 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:16.672253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:16.772703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:16.772671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:17.009011 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:17.008916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:17.289162 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:17.289047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:17.801367 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:17.801342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:20.066407 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:20.066356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:20.156185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:20.156155    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:21.212395 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.212341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:21.311394 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.311361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:21.453697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.453669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:21.679040 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.679011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:21.780004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.779972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:21.828536 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.828506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"]
Sep 07 16:40:21.834006 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.833989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:21.836798 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.836777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8ba4d4\""
Sep 07 16:40:21.836958 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.836936    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8ba4d4-dockercfg-nd6fz\""
Sep 07 16:40:21.842900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.842881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"]
Sep 07 16:40:21.908877 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.908844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:21.909072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:21.908894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vf4\" (UniqueName: \"kubernetes.io/projected/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-kube-api-access-k9vf4\") pod \"payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:22.010499 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:22.010404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:22.010499 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:22.010469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vf4\" (UniqueName: \"kubernetes.io/projected/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-kube-api-access-k9vf4\") pod \"payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:22.011175 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:22.011154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:22.030410 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:22.030385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vf4\" (UniqueName: \"kubernetes.io/projected/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-kube-api-access-k9vf4\") pod \"payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:22.144483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:22.144460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:22.414388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:22.414365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"]
Sep 07 16:40:22.416147 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:40:22.416117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded24a7e8_ce7c_40bd_a3cf_39f3e925226c.slice/crio-8d1eb54dd817ef83d82591cfe7f0141ece0f742bbfb5074a2cdcb5d78bfb454b WatchSource:0}: Error finding container 8d1eb54dd817ef83d82591cfe7f0141ece0f742bbfb5074a2cdcb5d78bfb454b: Status 404 returned error can't find the container with id 8d1eb54dd817ef83d82591cfe7f0141ece0f742bbfb5074a2cdcb5d78bfb454b
Sep 07 16:40:23.126777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:23.124833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd" event={"ID":"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c","Type":"ContainerStarted","Data":"e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a"}
Sep 07 16:40:23.126777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:23.124943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd" event={"ID":"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c","Type":"ContainerStarted","Data":"8d1eb54dd817ef83d82591cfe7f0141ece0f742bbfb5074a2cdcb5d78bfb454b"}
Sep 07 16:40:23.126777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:23.126052    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:23.141391 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:23.141334    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd" podStartSLOduration=2.141316883 podStartE2EDuration="2.141316883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:23.140523245 +0000 UTC m=+1228.022060213" watchObservedRunningTime="2026-09-07 16:40:23.141316883 +0000 UTC m=+1228.022853901"
Sep 07 16:40:31.872587 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:31.872509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:32.849804 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:32.849768    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"]
Sep 07 16:40:32.863853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:32.863827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:35.137922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:35.137896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:40:44.985185 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:44.985146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:45.231993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.231954    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"]
Sep 07 16:40:45.236002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.235977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.240071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.240047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"openshift-service-ca.crt\""
Sep 07 16:40:45.240071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.240084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"default-dockercfg-qcns6\""
Sep 07 16:40:45.240359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.240200    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"kube-root-ca.crt\""
Sep 07 16:40:45.240359 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.240251    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8d59e2\"/\"rate-test-model-8d59e2-kserve-self-signed-certs\""
Sep 07 16:40:45.246802 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.246774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"]
Sep 07 16:40:45.261925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.261898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:45.339375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.339333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglnk\" (UniqueName: \"kubernetes.io/projected/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-kube-api-access-lglnk\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.339375 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.339379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tls-certs\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.339600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.339407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-dshm\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.339600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.339464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tmp-dir\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.339600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.339522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-home\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.339600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.339559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-model-cache\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441131 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tmp-dir\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-home\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-model-cache\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglnk\" (UniqueName: \"kubernetes.io/projected/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-kube-api-access-lglnk\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tls-certs\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-dshm\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tmp-dir\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441683 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-home\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.441819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.441795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-model-cache\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.443922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.443899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-dshm\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.444413 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.444392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tls-certs\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.455051 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.455021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglnk\" (UniqueName: \"kubernetes.io/projected/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-kube-api-access-lglnk\") pod \"rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") " pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.496876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.496798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:45.547962 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.547926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:45.729692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.729653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:45.823910 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.823882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"]
Sep 07 16:40:45.825740 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:40:45.825713    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aae56a2_13c6_486a_8f28_ea4d4bc6c8a3.slice/crio-9c5b299532739d9fd0f0096ee0f3be660d95ffb1f7a022f0229b1bde7af61a11 WatchSource:0}: Error finding container 9c5b299532739d9fd0f0096ee0f3be660d95ffb1f7a022f0229b1bde7af61a11: Status 404 returned error can't find the container with id 9c5b299532739d9fd0f0096ee0f3be660d95ffb1f7a022f0229b1bde7af61a11
Sep 07 16:40:45.975546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:45.975502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:46.249446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:46.249415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v" event={"ID":"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3","Type":"ContainerStarted","Data":"c5e892ec24f1c8a424cf1d7f5e61d65ebdb041cb7d2d80c44e41e61facbc1517"}
Sep 07 16:40:46.249868 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:46.249458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v" event={"ID":"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3","Type":"ContainerStarted","Data":"9c5b299532739d9fd0f0096ee0f3be660d95ffb1f7a022f0229b1bde7af61a11"}
Sep 07 16:40:46.266282 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:46.266229    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v" podStartSLOduration=1.266211481 podStartE2EDuration="1.266211481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:46.264019923 +0000 UTC m=+1251.145556893" watchObservedRunningTime="2026-09-07 16:40:46.266211481 +0000 UTC m=+1251.147748449"
Sep 07 16:40:47.870482 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:47.870382    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr" podUID="9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" containerName="main" containerID="cri-o://dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a" gracePeriod=45
Sep 07 16:40:48.123843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.123786    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:40:48.261074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.261038    2577 generic.go:356] "Generic (PLEG): container finished" podID="9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" containerID="dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a" exitCode=0
Sep 07 16:40:48.261246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.261087    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"
Sep 07 16:40:48.261246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.261131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr" event={"ID":"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451","Type":"ContainerDied","Data":"dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a"}
Sep 07 16:40:48.261246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.261173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr" event={"ID":"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451","Type":"ContainerDied","Data":"943fb696e4e6886d95539262cc00212da4be0d16f56b51d280364707e05ebb9c"}
Sep 07 16:40:48.261246 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.261195    2577 scope.go:117] "RemoveContainer" containerID="dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a"
Sep 07 16:40:48.269946 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.269927    2577 scope.go:117] "RemoveContainer" containerID="dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a"
Sep 07 16:40:48.270225 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:40:48.270208    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a\": container with ID starting with dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a not found: ID does not exist" containerID="dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a"
Sep 07 16:40:48.270276 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.270235    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a"} err="failed to get container status \"dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a\": rpc error: code = NotFound desc = could not find container \"dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a\": container with ID starting with dec9edaa7e756bf838737af4b1107ac26a353f6ebe0c56b6439568068300247a not found: ID does not exist"
Sep 07 16:40:48.271494 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271477    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tmp-dir\") pod \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") "
Sep 07 16:40:48.271561 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpxs\" (UniqueName: \"kubernetes.io/projected/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-kube-api-access-btpxs\") pod \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") "
Sep 07 16:40:48.271602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tls-certs\") pod \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") "
Sep 07 16:40:48.271602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271591    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-home\") pod \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") "
Sep 07 16:40:48.271696 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-dshm\") pod \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") "
Sep 07 16:40:48.271749 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271734    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-model-cache\") pod \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\" (UID: \"9cfc5530-13da-4ca1-8ecb-7b7d1f25d451\") "
Sep 07 16:40:48.271749 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" (UID: "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:40:48.271927 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-home" (OuterVolumeSpecName: "home") pod "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" (UID: "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:40:48.271993 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.271956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" (UID: "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:40:48.272223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.272177    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:40:48.272223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.272197    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:40:48.272223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.272205    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:40:48.273842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.273819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" (UID: "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:40:48.273925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.273844    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-kube-api-access-btpxs" (OuterVolumeSpecName: "kube-api-access-btpxs") pod "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" (UID: "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451"). InnerVolumeSpecName "kube-api-access-btpxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:40:48.273925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.273818    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-dshm" (OuterVolumeSpecName: "dshm") pod "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" (UID: "9cfc5530-13da-4ca1-8ecb-7b7d1f25d451"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:40:48.373082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.373049    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpxs\" (UniqueName: \"kubernetes.io/projected/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-kube-api-access-btpxs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:40:48.373082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.373078    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:40:48.373082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.373089    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:40:48.582318 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.582283    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"]
Sep 07 16:40:48.586219 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:48.586193    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/sub-test-model-7aca0f-kserve-54967bdc47-26dzr"]
Sep 07 16:40:49.235281 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:49.235245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:49.333653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:49.333620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:49.568279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:49.568178    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" path="/var/lib/kubelet/pods/9cfc5530-13da-4ca1-8ecb-7b7d1f25d451/volumes"
Sep 07 16:40:50.381980 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:50.381944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:50.441157 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:50.441129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:50.774655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:50.774625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:50.875630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:50.875564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:50.973372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:50.973337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:51.152492 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.152462    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"]
Sep 07 16:40:51.152983 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.152967    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" containerName="main"
Sep 07 16:40:51.152983 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.152983    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" containerName="main"
Sep 07 16:40:51.153125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.153055    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cfc5530-13da-4ca1-8ecb-7b7d1f25d451" containerName="main"
Sep 07 16:40:51.157329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.157313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.159719 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.159697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d7b6a5\""
Sep 07 16:40:51.159850 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.159697    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d7b6a5-dockercfg-h9pbd\""
Sep 07 16:40:51.166992 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.166969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"]
Sep 07 16:40:51.301161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.301122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3b57b0-0493-40e8-a185-063c6f82555c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.301343 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.301258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr98\" (UniqueName: \"kubernetes.io/projected/ae3b57b0-0493-40e8-a185-063c6f82555c-kube-api-access-bwr98\") pod \"payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.402817 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.402699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr98\" (UniqueName: \"kubernetes.io/projected/ae3b57b0-0493-40e8-a185-063c6f82555c-kube-api-access-bwr98\") pod \"payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.403234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.402826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3b57b0-0493-40e8-a185-063c6f82555c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.403470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.403450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3b57b0-0493-40e8-a185-063c6f82555c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.410725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.410696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr98\" (UniqueName: \"kubernetes.io/projected/ae3b57b0-0493-40e8-a185-063c6f82555c-kube-api-access-bwr98\") pod \"payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.469320 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.469284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:51.749910 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:51.749887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"]
Sep 07 16:40:51.751904 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:40:51.751872    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3b57b0_0493_40e8_a185_063c6f82555c.slice/crio-9832ffcbe158aee9fa81468edd8cf8c7d4fdaf04a3439f005afa1f53544c16e8 WatchSource:0}: Error finding container 9832ffcbe158aee9fa81468edd8cf8c7d4fdaf04a3439f005afa1f53544c16e8: Status 404 returned error can't find the container with id 9832ffcbe158aee9fa81468edd8cf8c7d4fdaf04a3439f005afa1f53544c16e8
Sep 07 16:40:52.287179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:52.287139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7" event={"ID":"ae3b57b0-0493-40e8-a185-063c6f82555c","Type":"ContainerStarted","Data":"5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149"}
Sep 07 16:40:52.287365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:52.287199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7" event={"ID":"ae3b57b0-0493-40e8-a185-063c6f82555c","Type":"ContainerStarted","Data":"9832ffcbe158aee9fa81468edd8cf8c7d4fdaf04a3439f005afa1f53544c16e8"}
Sep 07 16:40:52.287365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:52.287225    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:40:52.306095 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:52.306036    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7" podStartSLOduration=1.306016177 podStartE2EDuration="1.306016177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:52.305867985 +0000 UTC m=+1257.187404955" watchObservedRunningTime="2026-09-07 16:40:52.306016177 +0000 UTC m=+1257.187553146"
Sep 07 16:40:55.548197 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:55.548164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:55.548602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:55.548216    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:55.560887 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:55.560858    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:56.320647 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:56.320620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:40:58.353035 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:58.352961    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:40:58.403928 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:40:58.403893    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:03.295717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:03.295679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:41:10.257285 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:10.257191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:11.276379 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:11.276344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:11.994194 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:11.994161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:14.154840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:14.154786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:16.901765 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:16.901721    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"]
Sep 07 16:41:16.906900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:16.906880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:16.910301 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:16.910276    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:41:16.912176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:16.912151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"]
Sep 07 16:41:16.996039 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:16.995954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:17.074882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:17.074847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d103407-2781-4ad0-82a7-f211472804ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl\" (UID: \"6d103407-2781-4ad0-82a7-f211472804ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:17.176218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:17.176122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d103407-2781-4ad0-82a7-f211472804ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl\" (UID: \"6d103407-2781-4ad0-82a7-f211472804ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:17.176778 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:17.176737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d103407-2781-4ad0-82a7-f211472804ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl\" (UID: \"6d103407-2781-4ad0-82a7-f211472804ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:17.219828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:17.219804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:17.492739 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:17.492711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"]
Sep 07 16:41:17.493966 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:41:17.493941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d103407_2781_4ad0_82a7_f211472804ec.slice/crio-07a8e180bf7cb91b0de3661195b21258458c1b6aad2471ed5afba77db7f7b4b9 WatchSource:0}: Error finding container 07a8e180bf7cb91b0de3661195b21258458c1b6aad2471ed5afba77db7f7b4b9: Status 404 returned error can't find the container with id 07a8e180bf7cb91b0de3661195b21258458c1b6aad2471ed5afba77db7f7b4b9
Sep 07 16:41:18.431404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:18.431373    2577 generic.go:356] "Generic (PLEG): container finished" podID="6d103407-2781-4ad0-82a7-f211472804ec" containerID="5f4b93a6916d170795b28e2dacc4a89314d84050bc5b8bd1f34fe0b123d90495" exitCode=0
Sep 07 16:41:18.431793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:18.431461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl" event={"ID":"6d103407-2781-4ad0-82a7-f211472804ec","Type":"ContainerDied","Data":"5f4b93a6916d170795b28e2dacc4a89314d84050bc5b8bd1f34fe0b123d90495"}
Sep 07 16:41:18.431793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:18.431508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl" event={"ID":"6d103407-2781-4ad0-82a7-f211472804ec","Type":"ContainerStarted","Data":"07a8e180bf7cb91b0de3661195b21258458c1b6aad2471ed5afba77db7f7b4b9"}
Sep 07 16:41:19.573634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:19.573610    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:19.702176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:19.702072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d103407-2781-4ad0-82a7-f211472804ec-maas-api-service-ca\") pod \"6d103407-2781-4ad0-82a7-f211472804ec\" (UID: \"6d103407-2781-4ad0-82a7-f211472804ec\") "
Sep 07 16:41:19.702462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:19.702438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d103407-2781-4ad0-82a7-f211472804ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d103407-2781-4ad0-82a7-f211472804ec" (UID: "6d103407-2781-4ad0-82a7-f211472804ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:41:19.803668 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:19.803631    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d103407-2781-4ad0-82a7-f211472804ec-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:41:20.444266 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:20.444228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl" event={"ID":"6d103407-2781-4ad0-82a7-f211472804ec","Type":"ContainerDied","Data":"07a8e180bf7cb91b0de3661195b21258458c1b6aad2471ed5afba77db7f7b4b9"}
Sep 07 16:41:20.444438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:20.444284    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a8e180bf7cb91b0de3661195b21258458c1b6aad2471ed5afba77db7f7b4b9"
Sep 07 16:41:20.444438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:20.444282    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"
Sep 07 16:41:22.170605 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:22.170570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:31.535384 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.535349    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"]
Sep 07 16:41:31.536070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.536053    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d103407-2781-4ad0-82a7-f211472804ec" containerName="revoke-keys"
Sep 07 16:41:31.536128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.536073    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d103407-2781-4ad0-82a7-f211472804ec" containerName="revoke-keys"
Sep 07 16:41:31.536164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.536138    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d103407-2781-4ad0-82a7-f211472804ec" containerName="revoke-keys"
Sep 07 16:41:31.539366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.539348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.543149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.543125    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"kube-root-ca.crt\""
Sep 07 16:41:31.543289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.543168    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"openshift-service-ca.crt\""
Sep 07 16:41:31.543289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.543134    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"rate-test-model-7aca0f-kserve-self-signed-certs\""
Sep 07 16:41:31.543289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.543134    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7aca0f\"/\"default-dockercfg-d4jx5\""
Sep 07 16:41:31.549206 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.549183    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"]
Sep 07 16:41:31.555495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.555464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:31.619253 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.619221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-tmp-dir\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.619438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.619266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-home\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.619438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.619289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9116adb-a864-419c-915e-3d971514f70b-tls-certs\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.619438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.619391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-dshm\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.619563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.619468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzprm\" (UniqueName: \"kubernetes.io/projected/a9116adb-a864-419c-915e-3d971514f70b-kube-api-access-jzprm\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.619563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.619488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-model-cache\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720323 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9116adb-a864-419c-915e-3d971514f70b-tls-certs\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-dshm\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzprm\" (UniqueName: \"kubernetes.io/projected/a9116adb-a864-419c-915e-3d971514f70b-kube-api-access-jzprm\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-model-cache\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720495 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-tmp-dir\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720690 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-home\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.720901 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-home\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.721012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-tmp-dir\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.721012 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.720957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-model-cache\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.722767 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.722736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-dshm\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.723115 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.723098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9116adb-a864-419c-915e-3d971514f70b-tls-certs\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.730740 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.730715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzprm\" (UniqueName: \"kubernetes.io/projected/a9116adb-a864-419c-915e-3d971514f70b-kube-api-access-jzprm\") pod \"rate-test-model-7aca0f-kserve-78c746bcff-b9g6d\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") " pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.850148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.850056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:31.947387 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:31.947342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:32.122907 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.122884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"]
Sep 07 16:41:32.125128 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:41:32.125092    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9116adb_a864_419c_915e_3d971514f70b.slice/crio-6c80e0b017b82d235a839d89137e826407f42a65debc29f91dfa1fb35c7cd3b8 WatchSource:0}: Error finding container 6c80e0b017b82d235a839d89137e826407f42a65debc29f91dfa1fb35c7cd3b8: Status 404 returned error can't find the container with id 6c80e0b017b82d235a839d89137e826407f42a65debc29f91dfa1fb35c7cd3b8
Sep 07 16:41:32.513056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.513021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:32.515381 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.515354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d" event={"ID":"a9116adb-a864-419c-915e-3d971514f70b","Type":"ContainerStarted","Data":"e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5"}
Sep 07 16:41:32.515533 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.515389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d" event={"ID":"a9116adb-a864-419c-915e-3d971514f70b","Type":"ContainerStarted","Data":"6c80e0b017b82d235a839d89137e826407f42a65debc29f91dfa1fb35c7cd3b8"}
Sep 07 16:41:32.532141 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.532093    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d" podStartSLOduration=1.5320779820000001 podStartE2EDuration="1.532077982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:41:32.531033016 +0000 UTC m=+1297.412569984" watchObservedRunningTime="2026-09-07 16:41:32.532077982 +0000 UTC m=+1297.413615000"
Sep 07 16:41:32.551217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.551187    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"]
Sep 07 16:41:32.551655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.551418    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd" podUID="ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" containerName="payload-processing" containerID="cri-o://e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a" gracePeriod=30
Sep 07 16:41:32.592614 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.592582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:32.653876 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.653844    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:32.760426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.760349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:32.760694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.760638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"]
Sep 07 16:41:32.763994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.763919    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ba4d4-f4xgl"]
Sep 07 16:41:32.816302 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.816277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:41:32.833592 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.833559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-plugins-config-volume\") pod \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") "
Sep 07 16:41:32.833704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.833601    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9vf4\" (UniqueName: \"kubernetes.io/projected/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-kube-api-access-k9vf4\") pod \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\" (UID: \"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c\") "
Sep 07 16:41:32.833936 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.833914    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" (UID: "ed24a7e8-ce7c-40bd-a3cf-39f3e925226c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:41:32.835825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.835780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:32.835964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.835943    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-kube-api-access-k9vf4" (OuterVolumeSpecName: "kube-api-access-k9vf4") pod "ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" (UID: "ed24a7e8-ce7c-40bd-a3cf-39f3e925226c"). InnerVolumeSpecName "kube-api-access-k9vf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:41:32.934740 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.934706    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:41:32.934740 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:32.934733    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vf4\" (UniqueName: \"kubernetes.io/projected/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c-kube-api-access-k9vf4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:41:33.107838 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.107700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:33.530694 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.530647    2577 generic.go:356] "Generic (PLEG): container finished" podID="ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" containerID="e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a" exitCode=0
Sep 07 16:41:33.530918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.530705    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"
Sep 07 16:41:33.530918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.530737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd" event={"ID":"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c","Type":"ContainerDied","Data":"e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a"}
Sep 07 16:41:33.530918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.530825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd" event={"ID":"ed24a7e8-ce7c-40bd-a3cf-39f3e925226c","Type":"ContainerDied","Data":"8d1eb54dd817ef83d82591cfe7f0141ece0f742bbfb5074a2cdcb5d78bfb454b"}
Sep 07 16:41:33.530918 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.530854    2577 scope.go:117] "RemoveContainer" containerID="e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a"
Sep 07 16:41:33.540717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.540697    2577 scope.go:117] "RemoveContainer" containerID="e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a"
Sep 07 16:41:33.541005 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:41:33.540988    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a\": container with ID starting with e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a not found: ID does not exist" containerID="e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a"
Sep 07 16:41:33.541067 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.541011    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a"} err="failed to get container status \"e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a\": rpc error: code = NotFound desc = could not find container \"e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a\": container with ID starting with e64761591a3d224dc4f8a3e8eb6ae63984f28c7080827b5ba2a56db6f4f80d7a not found: ID does not exist"
Sep 07 16:41:33.558742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.558713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"]
Sep 07 16:41:33.568326 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.568293    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d103407-2781-4ad0-82a7-f211472804ec" path="/var/lib/kubelet/pods/6d103407-2781-4ad0-82a7-f211472804ec/volumes"
Sep 07 16:41:33.568603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:33.568590    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ba4d4-679ddb5589-6zdqd"]
Sep 07 16:41:35.568664 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:35.568633    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" path="/var/lib/kubelet/pods/ed24a7e8-ce7c-40bd-a3cf-39f3e925226c/volumes"
Sep 07 16:41:39.088094 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:39.088055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:39.191294 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:39.191261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:39.363399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:39.363302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:39.766143 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:39.766106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:40.467232 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:40.467201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:41.061291 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:41.061257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:41.851259 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:41.851223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:41.851722 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:41.851281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:41.863689 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:41.863666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:42.595175 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:42.595150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:41:55.197145 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.197115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:55.224260 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.224221    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"]
Sep 07 16:41:55.224541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.224509    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7" podUID="ae3b57b0-0493-40e8-a185-063c6f82555c" containerName="payload-processing" containerID="cri-o://5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149" gracePeriod=30
Sep 07 16:41:55.296629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.296601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:55.372961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.372930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:55.430113 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.430082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:55.488304 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.488275    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:41:55.655841 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.655813    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae3b57b0-0493-40e8-a185-063c6f82555c" containerID="5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149" exitCode=0
Sep 07 16:41:55.656020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.655855    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"
Sep 07 16:41:55.656020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.655905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7" event={"ID":"ae3b57b0-0493-40e8-a185-063c6f82555c","Type":"ContainerDied","Data":"5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149"}
Sep 07 16:41:55.656020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.655960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7" event={"ID":"ae3b57b0-0493-40e8-a185-063c6f82555c","Type":"ContainerDied","Data":"9832ffcbe158aee9fa81468edd8cf8c7d4fdaf04a3439f005afa1f53544c16e8"}
Sep 07 16:41:55.656020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.655988    2577 scope.go:117] "RemoveContainer" containerID="5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149"
Sep 07 16:41:55.664952 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.664932    2577 scope.go:117] "RemoveContainer" containerID="5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149"
Sep 07 16:41:55.665217 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:41:55.665200    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149\": container with ID starting with 5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149 not found: ID does not exist" containerID="5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149"
Sep 07 16:41:55.665283 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.665224    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149"} err="failed to get container status \"5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149\": rpc error: code = NotFound desc = could not find container \"5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149\": container with ID starting with 5b98034ef50d68a4c3748b7cefcde8d67c1bc084698813783b586f8b54164149 not found: ID does not exist"
Sep 07 16:41:55.669502 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.669484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwr98\" (UniqueName: \"kubernetes.io/projected/ae3b57b0-0493-40e8-a185-063c6f82555c-kube-api-access-bwr98\") pod \"ae3b57b0-0493-40e8-a185-063c6f82555c\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") "
Sep 07 16:41:55.669637 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.669623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3b57b0-0493-40e8-a185-063c6f82555c-plugins-config-volume\") pod \"ae3b57b0-0493-40e8-a185-063c6f82555c\" (UID: \"ae3b57b0-0493-40e8-a185-063c6f82555c\") "
Sep 07 16:41:55.670000 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.669977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3b57b0-0493-40e8-a185-063c6f82555c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae3b57b0-0493-40e8-a185-063c6f82555c" (UID: "ae3b57b0-0493-40e8-a185-063c6f82555c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:41:55.671622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.671597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3b57b0-0493-40e8-a185-063c6f82555c-kube-api-access-bwr98" (OuterVolumeSpecName: "kube-api-access-bwr98") pod "ae3b57b0-0493-40e8-a185-063c6f82555c" (UID: "ae3b57b0-0493-40e8-a185-063c6f82555c"). InnerVolumeSpecName "kube-api-access-bwr98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:41:55.710895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.710808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:41:55.743330 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.743297    2577 scope.go:117] "RemoveContainer" containerID="1bfda06fae85ffae84a47210895c3c238b5280bb39f13ca35d613f09f0a77887"
Sep 07 16:41:55.752467 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.752443    2577 scope.go:117] "RemoveContainer" containerID="976f37fd9e6264beae432028e61dc26a7d4e3ec371871c2238a53657b8b4cae2"
Sep 07 16:41:55.761014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.760980    2577 scope.go:117] "RemoveContainer" containerID="2e4ac1c575bbf29bedf584b515f7e91ea44c73a35ec905a5fb2b19553cabf23a"
Sep 07 16:41:55.771224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.771195    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3b57b0-0493-40e8-a185-063c6f82555c-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:41:55.771224 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.771224    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwr98\" (UniqueName: \"kubernetes.io/projected/ae3b57b0-0493-40e8-a185-063c6f82555c-kube-api-access-bwr98\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:41:55.976800 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.976767    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"]
Sep 07 16:41:55.980788 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:55.980747    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7b6a5-6fc9fd5979-bpfk7"]
Sep 07 16:41:57.568691 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:57.568656    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3b57b0-0493-40e8-a185-063c6f82555c" path="/var/lib/kubelet/pods/ae3b57b0-0493-40e8-a185-063c6f82555c/volumes"
Sep 07 16:41:59.530693 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:41:59.530663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:01.363488 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:01.363456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:01.462725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:01.462688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:01.597160 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:01.597129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:02.170915 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:02.170885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:12.453705 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.453638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:12.486395 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.486358    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"]
Sep 07 16:42:12.486628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.486602    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" podUID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" containerName="payload-pre-processing" containerID="cri-o://a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e" gracePeriod=30
Sep 07 16:42:12.554149 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.554118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:12.649771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.649718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:12.706471 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.706391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:12.739120 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.739096    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:42:12.754355 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.754324    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" containerID="a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e" exitCode=0
Sep 07 16:42:12.754527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.754496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" event={"ID":"8bc084ea-5dcf-401d-ba91-52d602a4ed57","Type":"ContainerDied","Data":"a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e"}
Sep 07 16:42:12.754628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.754560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" event={"ID":"8bc084ea-5dcf-401d-ba91-52d602a4ed57","Type":"ContainerDied","Data":"bcb8d7a32b528a00ead3927f26f4b9b958f00d1ec515cf1caccd63705242f7c7"}
Sep 07 16:42:12.754628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.754570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"
Sep 07 16:42:12.754628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.754589    2577 scope.go:117] "RemoveContainer" containerID="a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e"
Sep 07 16:42:12.766468 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.766449    2577 scope.go:117] "RemoveContainer" containerID="a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e"
Sep 07 16:42:12.766793 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:42:12.766768    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e\": container with ID starting with a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e not found: ID does not exist" containerID="a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e"
Sep 07 16:42:12.766895 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.766802    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e"} err="failed to get container status \"a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e\": rpc error: code = NotFound desc = could not find container \"a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e\": container with ID starting with a262152be4db9719e0144a2c51b2d59a11585f84fba43fa8c417566139bf8f6e not found: ID does not exist"
Sep 07 16:42:12.841924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.841842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc084ea-5dcf-401d-ba91-52d602a4ed57-plugins-config-volume\") pod \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") "
Sep 07 16:42:12.842082 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.842046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm24d\" (UniqueName: \"kubernetes.io/projected/8bc084ea-5dcf-401d-ba91-52d602a4ed57-kube-api-access-pm24d\") pod \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\" (UID: \"8bc084ea-5dcf-401d-ba91-52d602a4ed57\") "
Sep 07 16:42:12.842228 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.842204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc084ea-5dcf-401d-ba91-52d602a4ed57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bc084ea-5dcf-401d-ba91-52d602a4ed57" (UID: "8bc084ea-5dcf-401d-ba91-52d602a4ed57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:42:12.842414 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.842396    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc084ea-5dcf-401d-ba91-52d602a4ed57-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:12.844350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.844327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc084ea-5dcf-401d-ba91-52d602a4ed57-kube-api-access-pm24d" (OuterVolumeSpecName: "kube-api-access-pm24d") pod "8bc084ea-5dcf-401d-ba91-52d602a4ed57" (UID: "8bc084ea-5dcf-401d-ba91-52d602a4ed57"). InnerVolumeSpecName "kube-api-access-pm24d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:12.943520 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.943488    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm24d\" (UniqueName: \"kubernetes.io/projected/8bc084ea-5dcf-401d-ba91-52d602a4ed57-kube-api-access-pm24d\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:12.977725 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:12.977646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:13.076345 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:13.076306    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"]
Sep 07 16:42:13.078400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:13.078381    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px"]
Sep 07 16:42:13.568229 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:13.568199    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" path="/var/lib/kubelet/pods/8bc084ea-5dcf-401d-ba91-52d602a4ed57/volumes"
Sep 07 16:42:13.670398 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:13.670362    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd6c5a-65cdf988c8-9k4px" podUID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.60:9004: i/o timeout"
Sep 07 16:42:18.735426 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:18.735344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:18.799144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:18.799113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:19.020794 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:19.020684    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"]
Sep 07 16:42:19.021095 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:19.021061    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy" containerID="cri-o://f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc" gracePeriod=30
Sep 07 16:42:19.053991 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:19.053958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:19.427174 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:19.427145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:22.650442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:22.650408    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"]
Sep 07 16:42:22.670027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:22.669996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:23.520861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:23.520785    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"]
Sep 07 16:42:23.542553 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:23.542524    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:23.951684 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:23.951652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:24.225735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.225699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:24.287713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.287688    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:42:24.363692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-envoy\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.363692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-credential-socket\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.363692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-podinfo\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-certs\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d6ts\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-kube-api-access-4d6ts\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-data\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363859    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-token\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-socket\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364014 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.363961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a54fcc3-20df-4f2e-8d63-2728436f6156-istiod-ca-cert\") pod \"0a54fcc3-20df-4f2e-8d63-2728436f6156\" (UID: \"0a54fcc3-20df-4f2e-8d63-2728436f6156\") "
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364023    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364345    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364363    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:24.364822 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.364586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a54fcc3-20df-4f2e-8d63-2728436f6156-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:42:24.366343 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.366307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:24.366454 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.366373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:42:24.366454 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.366442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:24.366696 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.366667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-kube-api-access-4d6ts" (OuterVolumeSpecName: "kube-api-access-4d6ts") pod "0a54fcc3-20df-4f2e-8d63-2728436f6156" (UID: "0a54fcc3-20df-4f2e-8d63-2728436f6156"). InnerVolumeSpecName "kube-api-access-4d6ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:24.415388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.415342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465063    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a54fcc3-20df-4f2e-8d63-2728436f6156-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465099    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465107    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465115    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465124    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6ts\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-kube-api-access-4d6ts\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465133    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.465159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.465140    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a54fcc3-20df-4f2e-8d63-2728436f6156-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:24.819663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.819573    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerID="f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc" exitCode=0
Sep 07 16:42:24.819663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.819622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"
Sep 07 16:42:24.819663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.819636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" event={"ID":"0a54fcc3-20df-4f2e-8d63-2728436f6156","Type":"ContainerDied","Data":"f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc"}
Sep 07 16:42:24.819959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.819692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b" event={"ID":"0a54fcc3-20df-4f2e-8d63-2728436f6156","Type":"ContainerDied","Data":"b3b03d850eafc5971e419cee2bc679057d32de10606d864977b34b960a67220a"}
Sep 07 16:42:24.819959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.819719    2577 scope.go:117] "RemoveContainer" containerID="f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc"
Sep 07 16:42:24.829634 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.829615    2577 scope.go:117] "RemoveContainer" containerID="d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"
Sep 07 16:42:24.838041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.838020    2577 scope.go:117] "RemoveContainer" containerID="f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc"
Sep 07 16:42:24.838304 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:42:24.838286    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc\": container with ID starting with f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc not found: ID does not exist" containerID="f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc"
Sep 07 16:42:24.838350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.838314    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc"} err="failed to get container status \"f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc\": rpc error: code = NotFound desc = could not find container \"f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc\": container with ID starting with f5ccdd71c4023b77fe3e466a547db361e88d1953734d5b553ab8a0188a95efbc not found: ID does not exist"
Sep 07 16:42:24.838350 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.838330    2577 scope.go:117] "RemoveContainer" containerID="d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"
Sep 07 16:42:24.838582 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:42:24.838565    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26\": container with ID starting with d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26 not found: ID does not exist" containerID="d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"
Sep 07 16:42:24.838621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.838589    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26"} err="failed to get container status \"d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26\": rpc error: code = NotFound desc = could not find container \"d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26\": container with ID starting with d9ff29ae1e1046beb245a5df7e29cb883985d3cfcc1a7e4e04d1a38dea4eec26 not found: ID does not exist"
Sep 07 16:42:24.841514 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.841495    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"]
Sep 07 16:42:24.845767 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:24.845739    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd6c5a-openshift-default-6d74cdfcfb-jxk6b"]
Sep 07 16:42:25.568276 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:25.568234    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" path="/var/lib/kubelet/pods/0a54fcc3-20df-4f2e-8d63-2728436f6156/volumes"
Sep 07 16:42:27.731307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:27.731273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:27.830602 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:27.830565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:28.992546 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:28.992502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.093464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.093430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.193156 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.193118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.445336 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.445308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.512225 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.512190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.582153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582120    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"]
Sep 07 16:42:29.582526 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582515    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" containerName="payload-pre-processing"
Sep 07 16:42:29.582570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582529    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" containerName="payload-pre-processing"
Sep 07 16:42:29.582570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" containerName="payload-processing"
Sep 07 16:42:29.582570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" containerName="payload-processing"
Sep 07 16:42:29.582570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582555    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy"
Sep 07 16:42:29.582570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582561    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582572    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3b57b0-0493-40e8-a185-063c6f82555c" containerName="payload-processing"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582577    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3b57b0-0493-40e8-a185-063c6f82555c" containerName="payload-processing"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582591    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582661    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3b57b0-0493-40e8-a185-063c6f82555c" containerName="payload-processing"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582668    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed24a7e8-ce7c-40bd-a3cf-39f3e925226c" containerName="payload-processing"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582675    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc084ea-5dcf-401d-ba91-52d602a4ed57" containerName="payload-pre-processing"
Sep 07 16:42:29.582811 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.582681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy"
Sep 07 16:42:29.587120 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.587103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.589827 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.589807    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4180e8-dockercfg-mx4l9\""
Sep 07 16:42:29.589938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.589827    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4180e8\""
Sep 07 16:42:29.596152 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.596130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"]
Sep 07 16:42:29.721095 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.720999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ht2d\" (UniqueName: \"kubernetes.io/projected/01c751f9-ec80-4905-93a0-25e3b38a1c3a-kube-api-access-9ht2d\") pod \"payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.721387 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.721368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c751f9-ec80-4905-93a0-25e3b38a1c3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.747647 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.747612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.780861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.780829    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"]
Sep 07 16:42:29.781321 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.781290    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd" podUID="85408405-b3b1-4cf7-aa34-a19ee068886e" containerName="payload-pre-processing" containerID="cri-o://24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e" gracePeriod=30
Sep 07 16:42:29.822501 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.822458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ht2d\" (UniqueName: \"kubernetes.io/projected/01c751f9-ec80-4905-93a0-25e3b38a1c3a-kube-api-access-9ht2d\") pod \"payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.822679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.822566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c751f9-ec80-4905-93a0-25e3b38a1c3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.823251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.823228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c751f9-ec80-4905-93a0-25e3b38a1c3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.830466 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.830443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ht2d\" (UniqueName: \"kubernetes.io/projected/01c751f9-ec80-4905-93a0-25e3b38a1c3a-kube-api-access-9ht2d\") pod \"payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.847285 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.847252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:29.899392 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.899362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:29.951641 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:29.951204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:30.046341 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.046318    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:42:30.047096 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.047073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:30.171940 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.171918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"]
Sep 07 16:42:30.173841 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:42:30.173814    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c751f9_ec80_4905_93a0_25e3b38a1c3a.slice/crio-d3433255e0fead20d3d0e5185b166faf53170f3d80deac70540e9aaaf4b48f36 WatchSource:0}: Error finding container d3433255e0fead20d3d0e5185b166faf53170f3d80deac70540e9aaaf4b48f36: Status 404 returned error can't find the container with id d3433255e0fead20d3d0e5185b166faf53170f3d80deac70540e9aaaf4b48f36
Sep 07 16:42:30.225938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.225871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85408405-b3b1-4cf7-aa34-a19ee068886e-plugins-config-volume\") pod \"85408405-b3b1-4cf7-aa34-a19ee068886e\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") "
Sep 07 16:42:30.226159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.225982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwxc4\" (UniqueName: \"kubernetes.io/projected/85408405-b3b1-4cf7-aa34-a19ee068886e-kube-api-access-cwxc4\") pod \"85408405-b3b1-4cf7-aa34-a19ee068886e\" (UID: \"85408405-b3b1-4cf7-aa34-a19ee068886e\") "
Sep 07 16:42:30.226206 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.226190    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85408405-b3b1-4cf7-aa34-a19ee068886e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85408405-b3b1-4cf7-aa34-a19ee068886e" (UID: "85408405-b3b1-4cf7-aa34-a19ee068886e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:42:30.226336 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.226272    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85408405-b3b1-4cf7-aa34-a19ee068886e-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:30.228212 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.228187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85408405-b3b1-4cf7-aa34-a19ee068886e-kube-api-access-cwxc4" (OuterVolumeSpecName: "kube-api-access-cwxc4") pod "85408405-b3b1-4cf7-aa34-a19ee068886e" (UID: "85408405-b3b1-4cf7-aa34-a19ee068886e"). InnerVolumeSpecName "kube-api-access-cwxc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:30.270073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.270031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:30.327872 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.327834    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwxc4\" (UniqueName: \"kubernetes.io/projected/85408405-b3b1-4cf7-aa34-a19ee068886e-kube-api-access-cwxc4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:30.852428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.852392    2577 generic.go:356] "Generic (PLEG): container finished" podID="85408405-b3b1-4cf7-aa34-a19ee068886e" containerID="24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e" exitCode=0
Sep 07 16:42:30.852628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.852433    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"
Sep 07 16:42:30.852628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.852483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd" event={"ID":"85408405-b3b1-4cf7-aa34-a19ee068886e","Type":"ContainerDied","Data":"24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e"}
Sep 07 16:42:30.852628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.852539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd" event={"ID":"85408405-b3b1-4cf7-aa34-a19ee068886e","Type":"ContainerDied","Data":"62c235b601d40e6dd57ed0dd034234d983e19805883c4736c010fb425a099d4a"}
Sep 07 16:42:30.852628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.852562    2577 scope.go:117] "RemoveContainer" containerID="24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e"
Sep 07 16:42:30.854349 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.854326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g" event={"ID":"01c751f9-ec80-4905-93a0-25e3b38a1c3a","Type":"ContainerStarted","Data":"c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251"}
Sep 07 16:42:30.854460 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.854375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g" event={"ID":"01c751f9-ec80-4905-93a0-25e3b38a1c3a","Type":"ContainerStarted","Data":"d3433255e0fead20d3d0e5185b166faf53170f3d80deac70540e9aaaf4b48f36"}
Sep 07 16:42:30.854521 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.854467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:30.862633 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.862616    2577 scope.go:117] "RemoveContainer" containerID="24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e"
Sep 07 16:42:30.862942 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:42:30.862925    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e\": container with ID starting with 24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e not found: ID does not exist" containerID="24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e"
Sep 07 16:42:30.862995 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.862950    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e"} err="failed to get container status \"24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e\": rpc error: code = NotFound desc = could not find container \"24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e\": container with ID starting with 24e2ed4b25e11017acfc99169f0c18d9c66120abbb90a8bdb774d6e66fb35c7e not found: ID does not exist"
Sep 07 16:42:30.872238 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.872193    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g" podStartSLOduration=1.872181031 podStartE2EDuration="1.872181031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:42:30.870820343 +0000 UTC m=+1355.752357313" watchObservedRunningTime="2026-09-07 16:42:30.872181031 +0000 UTC m=+1355.753717998"
Sep 07 16:42:30.882230 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.882195    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"]
Sep 07 16:42:30.885483 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:30.885453    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4a9a21-6889b85cc8-kk6wd"]
Sep 07 16:42:31.574874 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:31.574844    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85408405-b3b1-4cf7-aa34-a19ee068886e" path="/var/lib/kubelet/pods/85408405-b3b1-4cf7-aa34-a19ee068886e/volumes"
Sep 07 16:42:36.552935 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:36.552899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:36.652098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:36.652064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:36.774122 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:36.774090    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"]
Sep 07 16:42:36.774358 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:36.774329    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy" containerID="cri-o://88617b06bc3575775a63df46e950932c6863da013973c2d8d939f05e53756573" gracePeriod=30
Sep 07 16:42:36.798315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:36.798277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:37.669921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.669873    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v" podUID="9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" containerName="main" containerID="cri-o://c5e892ec24f1c8a424cf1d7f5e61d65ebdb041cb7d2d80c44e41e61facbc1517" gracePeriod=45
Sep 07 16:42:37.895245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.895209    2577 generic.go:356] "Generic (PLEG): container finished" podID="9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" containerID="c5e892ec24f1c8a424cf1d7f5e61d65ebdb041cb7d2d80c44e41e61facbc1517" exitCode=0
Sep 07 16:42:37.895380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.895254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v" event={"ID":"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3","Type":"ContainerDied","Data":"c5e892ec24f1c8a424cf1d7f5e61d65ebdb041cb7d2d80c44e41e61facbc1517"}
Sep 07 16:42:37.914269 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.914246    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:42:37.999309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tmp-dir\") pod \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") "
Sep 07 16:42:37.999309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-dshm\") pod \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") "
Sep 07 16:42:37.999309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-home\") pod \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") "
Sep 07 16:42:37.999309 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-model-cache\") pod \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") "
Sep 07 16:42:37.999648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lglnk\" (UniqueName: \"kubernetes.io/projected/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-kube-api-access-lglnk\") pod \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") "
Sep 07 16:42:37.999648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tls-certs\") pod \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\" (UID: \"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3\") "
Sep 07 16:42:37.999648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-home" (OuterVolumeSpecName: "home") pod "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" (UID: "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:37.999648 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999534    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" (UID: "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:37.999885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999655    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:37.999885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999671    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:37.999885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:37.999659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" (UID: "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:38.001462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.001439    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-dshm" (OuterVolumeSpecName: "dshm") pod "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" (UID: "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:38.001608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.001590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-kube-api-access-lglnk" (OuterVolumeSpecName: "kube-api-access-lglnk") pod "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" (UID: "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3"). InnerVolumeSpecName "kube-api-access-lglnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:38.001608 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.001594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" (UID: "9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:42:38.100700 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.100661    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglnk\" (UniqueName: \"kubernetes.io/projected/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-kube-api-access-lglnk\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.100700 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.100693    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.100700 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.100704    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.100700 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.100712    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.547559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.547522    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d" podUID="a9116adb-a864-419c-915e-3d971514f70b" containerName="main" containerID="cri-o://e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5" gracePeriod=45
Sep 07 16:42:38.805519 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.805455    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:42:38.902257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.902225    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"
Sep 07 16:42:38.902434 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.902262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v" event={"ID":"9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3","Type":"ContainerDied","Data":"9c5b299532739d9fd0f0096ee0f3be660d95ffb1f7a022f0229b1bde7af61a11"}
Sep 07 16:42:38.902434 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.902310    2577 scope.go:117] "RemoveContainer" containerID="c5e892ec24f1c8a424cf1d7f5e61d65ebdb041cb7d2d80c44e41e61facbc1517"
Sep 07 16:42:38.903808 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.903783    2577 generic.go:356] "Generic (PLEG): container finished" podID="a9116adb-a864-419c-915e-3d971514f70b" containerID="e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5" exitCode=0
Sep 07 16:42:38.903925 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.903825    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"
Sep 07 16:42:38.903995 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.903824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d" event={"ID":"a9116adb-a864-419c-915e-3d971514f70b","Type":"ContainerDied","Data":"e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5"}
Sep 07 16:42:38.904053 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.904009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d" event={"ID":"a9116adb-a864-419c-915e-3d971514f70b","Type":"ContainerDied","Data":"6c80e0b017b82d235a839d89137e826407f42a65debc29f91dfa1fb35c7cd3b8"}
Sep 07 16:42:38.909154 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-model-cache\") pod \"a9116adb-a864-419c-915e-3d971514f70b\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") "
Sep 07 16:42:38.909283 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9116adb-a864-419c-915e-3d971514f70b-tls-certs\") pod \"a9116adb-a864-419c-915e-3d971514f70b\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") "
Sep 07 16:42:38.909283 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-home\") pod \"a9116adb-a864-419c-915e-3d971514f70b\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") "
Sep 07 16:42:38.909399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-dshm\") pod \"a9116adb-a864-419c-915e-3d971514f70b\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") "
Sep 07 16:42:38.909399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-tmp-dir\") pod \"a9116adb-a864-419c-915e-3d971514f70b\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") "
Sep 07 16:42:38.909399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzprm\" (UniqueName: \"kubernetes.io/projected/a9116adb-a864-419c-915e-3d971514f70b-kube-api-access-jzprm\") pod \"a9116adb-a864-419c-915e-3d971514f70b\" (UID: \"a9116adb-a864-419c-915e-3d971514f70b\") "
Sep 07 16:42:38.909532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9116adb-a864-419c-915e-3d971514f70b" (UID: "a9116adb-a864-419c-915e-3d971514f70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:38.909532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909443    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-home" (OuterVolumeSpecName: "home") pod "a9116adb-a864-419c-915e-3d971514f70b" (UID: "a9116adb-a864-419c-915e-3d971514f70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:38.909623 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909539    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9116adb-a864-419c-915e-3d971514f70b" (UID: "a9116adb-a864-419c-915e-3d971514f70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:38.909742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909726    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.909824 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909768    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.909824 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.909781    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:38.911949 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.911923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9116adb-a864-419c-915e-3d971514f70b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9116adb-a864-419c-915e-3d971514f70b" (UID: "a9116adb-a864-419c-915e-3d971514f70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:42:38.912039 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.911926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9116adb-a864-419c-915e-3d971514f70b-kube-api-access-jzprm" (OuterVolumeSpecName: "kube-api-access-jzprm") pod "a9116adb-a864-419c-915e-3d971514f70b" (UID: "a9116adb-a864-419c-915e-3d971514f70b"). InnerVolumeSpecName "kube-api-access-jzprm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:38.912241 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.912223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-dshm" (OuterVolumeSpecName: "dshm") pod "a9116adb-a864-419c-915e-3d971514f70b" (UID: "a9116adb-a864-419c-915e-3d971514f70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:38.912724 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.912701    2577 scope.go:117] "RemoveContainer" containerID="e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5"
Sep 07 16:42:38.930909 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.930889    2577 scope.go:117] "RemoveContainer" containerID="e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5"
Sep 07 16:42:38.931211 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:42:38.931193    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5\": container with ID starting with e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5 not found: ID does not exist" containerID="e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5"
Sep 07 16:42:38.931257 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.931220    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5"} err="failed to get container status \"e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5\": rpc error: code = NotFound desc = could not find container \"e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5\": container with ID starting with e9d30bba4b7dcd8aa0df01da165c12fe0666da5fe6a7268a0c342d42c6324eb5 not found: ID does not exist"
Sep 07 16:42:38.938401 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.938377    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"]
Sep 07 16:42:38.942177 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:38.942155    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8d59e2/rate-test-model-8d59e2-kserve-6ddc5f4587-6pm8v"]
Sep 07 16:42:39.010630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.010599    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9116adb-a864-419c-915e-3d971514f70b-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:39.010630 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.010624    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9116adb-a864-419c-915e-3d971514f70b-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:39.010799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.010640    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzprm\" (UniqueName: \"kubernetes.io/projected/a9116adb-a864-419c-915e-3d971514f70b-kube-api-access-jzprm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:39.224094 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.224062    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"]
Sep 07 16:42:39.227760 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.227725    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7aca0f/rate-test-model-7aca0f-kserve-78c746bcff-b9g6d"]
Sep 07 16:42:39.567764 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.567672    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" path="/var/lib/kubelet/pods/9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3/volumes"
Sep 07 16:42:39.568109 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:39.568095    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9116adb-a864-419c-915e-3d971514f70b" path="/var/lib/kubelet/pods/a9116adb-a864-419c-915e-3d971514f70b/volumes"
Sep 07 16:42:41.862885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:41.862856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:42:41.926652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:41.926621    2577 generic.go:356] "Generic (PLEG): container finished" podID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerID="88617b06bc3575775a63df46e950932c6863da013973c2d8d939f05e53756573" exitCode=0
Sep 07 16:42:41.926853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:41.926692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" event={"ID":"5975e495-336c-4993-8bf7-ae7ed658ff4a","Type":"ContainerDied","Data":"88617b06bc3575775a63df46e950932c6863da013973c2d8d939f05e53756573"}
Sep 07 16:42:41.926853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:41.926791    2577 scope.go:117] "RemoveContainer" containerID="abe47834c7ce96344cc321a8d8476075323c69c07716fe02abe885e4d2a732da"
Sep 07 16:42:42.034595 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.034571    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:42:42.139083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-token\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-credential-socket\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-podinfo\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg8xn\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-kube-api-access-mg8xn\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139218 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-data\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5975e495-336c-4993-8bf7-ae7ed658ff4a-istiod-ca-cert\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139458 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:42.139458 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-envoy\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139460    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:42.139558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-certs\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-socket\") pod \"5975e495-336c-4993-8bf7-ae7ed658ff4a\" (UID: \"5975e495-336c-4993-8bf7-ae7ed658ff4a\") "
Sep 07 16:42:42.139712 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:42.139801 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5975e495-336c-4993-8bf7-ae7ed658ff4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:42:42.139964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.139930    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:42.140198 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.140178    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.140234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.140206    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.140234 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.140220    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.140337 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.140234    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5975e495-336c-4993-8bf7-ae7ed658ff4a-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.141652 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.141622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:42:42.141771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.141677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:42.141771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.141680    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-kube-api-access-mg8xn" (OuterVolumeSpecName: "kube-api-access-mg8xn") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "kube-api-access-mg8xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:42:42.141771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.141724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5975e495-336c-4993-8bf7-ae7ed658ff4a" (UID: "5975e495-336c-4993-8bf7-ae7ed658ff4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:42:42.241107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.241073    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.241107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.241099    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.241107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.241110    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.241107 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.241117    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8xn\" (UniqueName: \"kubernetes.io/projected/5975e495-336c-4993-8bf7-ae7ed658ff4a-kube-api-access-mg8xn\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.241366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.241125    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5975e495-336c-4993-8bf7-ae7ed658ff4a-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:42:42.933189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.933159    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"
Sep 07 16:42:42.933639 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.933203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7" event={"ID":"5975e495-336c-4993-8bf7-ae7ed658ff4a","Type":"ContainerDied","Data":"d45911f0c1da70be2adcae8eff00a0841efb41b9a45f7a6ded1d771d4e5107bf"}
Sep 07 16:42:42.933639 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.933272    2577 scope.go:117] "RemoveContainer" containerID="88617b06bc3575775a63df46e950932c6863da013973c2d8d939f05e53756573"
Sep 07 16:42:42.954074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.954047    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"]
Sep 07 16:42:42.958015 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:42.957990    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a9a21-openshift-default-5c96cb8fdd-zvtl7"]
Sep 07 16:42:43.567851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:43.567821    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" path="/var/lib/kubelet/pods/5975e495-336c-4993-8bf7-ae7ed658ff4a/volumes"
Sep 07 16:42:52.673261 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:52.673230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:53.313715 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.313679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:53.570531 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570463    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"]
Sep 07 16:42:53.570881 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy"
Sep 07 16:42:53.570881 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570881    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570888    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9116adb-a864-419c-915e-3d971514f70b" containerName="main"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570893    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9116adb-a864-419c-915e-3d971514f70b" containerName="main"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570904    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85408405-b3b1-4cf7-aa34-a19ee068886e" containerName="payload-pre-processing"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570909    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85408405-b3b1-4cf7-aa34-a19ee068886e" containerName="payload-pre-processing"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570919    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" containerName="main"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570923    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" containerName="main"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570935    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570940    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.570999    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.571005    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85408405-b3b1-4cf7-aa34-a19ee068886e" containerName="payload-pre-processing"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.571012    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aae56a2-13c6-486a-8f28-ea4d4bc6c8a3" containerName="main"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.571018    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a54fcc3-20df-4f2e-8d63-2728436f6156" containerName="istio-proxy"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.571024    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9116adb-a864-419c-915e-3d971514f70b" containerName="main"
Sep 07 16:42:53.571056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.571030    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5975e495-336c-4993-8bf7-ae7ed658ff4a" containerName="istio-proxy"
Sep 07 16:42:53.575463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.575446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.578001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.577975    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6a45e8-openshift-default-dockercfg-8hskj\""
Sep 07 16:42:53.582553 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.582533    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"]
Sep 07 16:42:53.597454 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.597432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:53.652979 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.652932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-podinfo\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-certs\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-envoy\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istiod-ca-cert\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-data\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-token\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-socket\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653522 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kp2\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-kube-api-access-g7kp2\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.653522 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.653445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-credential-socket\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.754819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.754783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kp2\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-kube-api-access-g7kp2\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.754819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.754825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-credential-socket\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.754867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-podinfo\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.754890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-certs\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-envoy\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istiod-ca-cert\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-data\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-token\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-socket\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-certs\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-credential-socket\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-socket\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istiod-ca-cert\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.755766 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.755733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-data\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.757865 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.757846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-envoy\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.758162 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.758142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-podinfo\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.763363 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.763332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kp2\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-kube-api-access-g7kp2\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.763469 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.763339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-token\") pod \"e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") " pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:53.767264 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.767243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:53.888256 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:53.888222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:54.165029 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:54.164995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"]
Sep 07 16:42:54.166413 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:42:54.166384    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2e6379_9c61_4533_affd_b9e3e28f9e4d.slice/crio-a73958df0c61ebf7eb173231f2dfe02896e56d8087785dd5ce21eb0dddf561ce WatchSource:0}: Error finding container a73958df0c61ebf7eb173231f2dfe02896e56d8087785dd5ce21eb0dddf561ce: Status 404 returned error can't find the container with id a73958df0c61ebf7eb173231f2dfe02896e56d8087785dd5ce21eb0dddf561ce
Sep 07 16:42:54.168590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:54.168552    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:42:54.168701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:54.168621    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:42:54.168701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:54.168649    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:42:55.001009 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:55.000980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" event={"ID":"5e2e6379-9c61-4533-affd-b9e3e28f9e4d","Type":"ContainerStarted","Data":"5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"}
Sep 07 16:42:55.001401 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:55.001032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" event={"ID":"5e2e6379-9c61-4533-affd-b9e3e28f9e4d","Type":"ContainerStarted","Data":"a73958df0c61ebf7eb173231f2dfe02896e56d8087785dd5ce21eb0dddf561ce"}
Sep 07 16:42:55.020161 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:55.020114    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podStartSLOduration=2.020098373 podStartE2EDuration="2.020098373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:42:55.018772128 +0000 UTC m=+1379.900309094" watchObservedRunningTime="2026-09-07 16:42:55.020098373 +0000 UTC m=+1379.901635340"
Sep 07 16:42:55.286542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:55.286449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:55.889102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:55.889059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:55.893969 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:55.893946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:56.006169 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:56.006138    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:56.007207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:56.007187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:42:56.901950 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:56.901910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:57.022491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:57.022463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:58.065369 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.065338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:58.167517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.167486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:58.268071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.268023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:58.509988 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.509958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:58.609599 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.609567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:42:58.621718 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.621682    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"]
Sep 07 16:42:58.626900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.626874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.629975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.629955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a45e8-dockercfg-pgfk9\""
Sep 07 16:42:58.630136 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.629962    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a45e8\""
Sep 07 16:42:58.634619 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.634589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"]
Sep 07 16:42:58.704009 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.703971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09d3c6d-09aa-488b-afca-3107a68afc8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.704191 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.704095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsq5t\" (UniqueName: \"kubernetes.io/projected/d09d3c6d-09aa-488b-afca-3107a68afc8b-kube-api-access-zsq5t\") pod \"payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.805820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.805689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09d3c6d-09aa-488b-afca-3107a68afc8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.805820 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.805813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsq5t\" (UniqueName: \"kubernetes.io/projected/d09d3c6d-09aa-488b-afca-3107a68afc8b-kube-api-access-zsq5t\") pod \"payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.806422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.806400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09d3c6d-09aa-488b-afca-3107a68afc8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.813974 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.813937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsq5t\" (UniqueName: \"kubernetes.io/projected/d09d3c6d-09aa-488b-afca-3107a68afc8b-kube-api-access-zsq5t\") pod \"payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:58.940105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:58.940066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:42:59.208629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:42:59.208597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"]
Sep 07 16:42:59.211819 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:42:59.211788    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09d3c6d_09aa_488b_afca_3107a68afc8b.slice/crio-dee7ff8e5622b970b5254cf6eb681eda92b0123ede6555032ddd8cb2d2b10e72 WatchSource:0}: Error finding container dee7ff8e5622b970b5254cf6eb681eda92b0123ede6555032ddd8cb2d2b10e72: Status 404 returned error can't find the container with id dee7ff8e5622b970b5254cf6eb681eda92b0123ede6555032ddd8cb2d2b10e72
Sep 07 16:43:00.030490 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:00.030450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr" event={"ID":"d09d3c6d-09aa-488b-afca-3107a68afc8b","Type":"ContainerStarted","Data":"676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b"}
Sep 07 16:43:00.030685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:00.030510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr" event={"ID":"d09d3c6d-09aa-488b-afca-3107a68afc8b","Type":"ContainerStarted","Data":"dee7ff8e5622b970b5254cf6eb681eda92b0123ede6555032ddd8cb2d2b10e72"}
Sep 07 16:43:00.030685 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:00.030592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:43:00.046865 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:00.046815    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr" podStartSLOduration=2.046800336 podStartE2EDuration="2.046800336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:43:00.045420801 +0000 UTC m=+1384.926957764" watchObservedRunningTime="2026-09-07 16:43:00.046800336 +0000 UTC m=+1384.928337304"
Sep 07 16:43:11.037512 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:11.037475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:43:12.047939 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:12.047903    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 16:43:12.048322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:12.047977    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 16:43:14.102443 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:14.102412    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerID="5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6" exitCode=0
Sep 07 16:43:14.102873 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:14.102487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" event={"ID":"5e2e6379-9c61-4533-affd-b9e3e28f9e4d","Type":"ContainerDied","Data":"5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"}
Sep 07 16:43:14.102942 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:14.102928    2577 scope.go:117] "RemoveContainer" containerID="5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"
Sep 07 16:43:14.104143 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:14.104115    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:43:14.104209 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:14.104185    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:43:14.104245 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:14.104217    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:43:15.111606 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:15.111565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" event={"ID":"5e2e6379-9c61-4533-affd-b9e3e28f9e4d","Type":"ContainerStarted","Data":"664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15"}
Sep 07 16:43:15.888431 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:15.888390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:43:15.888697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:15.888676    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 16:43:15.888744 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:15.888727    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 16:43:16.871706 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:16.871669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:43:16.889572 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:16.889541    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 16:43:16.889747 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:16.889592    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 16:43:17.889075 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:17.889043    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 16:43:17.889454 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:17.889098    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 16:43:18.867269 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:18.867234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:43:18.889555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:18.889520    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 16:43:18.889972 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:18.889574    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 16:43:19.889464 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:19.889426    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 16:43:19.889646 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:19.889487    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 16:43:20.892356 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:20.892326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:43:20.892823 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:20.892606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:43:20.893301 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:20.893286    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:43:21.822189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:21.822150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:43:24.019369 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:24.019335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:43:24.120481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:24.120453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:43:30.231777 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.231705    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"]
Sep 07 16:43:30.236328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.236304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.239015 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.238997    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4180e8\"/\"kube-root-ca.crt\""
Sep 07 16:43:30.240059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.240040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4180e8\"/\"ipp-route-4180e8-kserve-self-signed-certs\""
Sep 07 16:43:30.240140 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.240041    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4180e8\"/\"default-dockercfg-nllzk\""
Sep 07 16:43:30.240182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.240048    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4180e8\"/\"openshift-service-ca.crt\""
Sep 07 16:43:30.245197 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.245170    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"]
Sep 07 16:43:30.254380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.254347    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:43:30.345785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.345730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-tmp-dir\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.345963 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.345818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-home\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.345963 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.345927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-model-cache\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.345963 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.345955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3062207-d164-4a7a-b7de-f4a4ad38c875-tls-certs\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.346074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.345985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splsm\" (UniqueName: \"kubernetes.io/projected/f3062207-d164-4a7a-b7de-f4a4ad38c875-kube-api-access-splsm\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.346074 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.346057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-dshm\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.447610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.447568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-model-cache\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.447610 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.447610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3062207-d164-4a7a-b7de-f4a4ad38c875-tls-certs\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.447877 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.447635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splsm\" (UniqueName: \"kubernetes.io/projected/f3062207-d164-4a7a-b7de-f4a4ad38c875-kube-api-access-splsm\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.447877 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.447661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-dshm\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.447877 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.447690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-tmp-dir\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.447877 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.447733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-home\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.448101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.448083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-model-cache\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.448165 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.448147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-tmp-dir\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.448204 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.448182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-home\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.450147 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.450120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-dshm\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.450368 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.450350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3062207-d164-4a7a-b7de-f4a4ad38c875-tls-certs\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.454904 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.454885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-splsm\" (UniqueName: \"kubernetes.io/projected/f3062207-d164-4a7a-b7de-f4a4ad38c875-kube-api-access-splsm\") pod \"ipp-route-4180e8-kserve-6bbfdbb6f9-92l69\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") " pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.547072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.546970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:30.825595 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:30.825561    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"]
Sep 07 16:43:31.208788 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:31.208724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69" event={"ID":"f3062207-d164-4a7a-b7de-f4a4ad38c875","Type":"ContainerStarted","Data":"038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8"}
Sep 07 16:43:31.208788 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:31.208789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69" event={"ID":"f3062207-d164-4a7a-b7de-f4a4ad38c875","Type":"ContainerStarted","Data":"b7a7972a4b01334d88d5474a81238141897e0a8d934c0640c36c22502fa7f8ec"}
Sep 07 16:43:31.224710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:31.224656    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69" podStartSLOduration=1.224637773 podStartE2EDuration="1.224637773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:43:31.223489667 +0000 UTC m=+1416.105026636" watchObservedRunningTime="2026-09-07 16:43:31.224637773 +0000 UTC m=+1416.106174740"
Sep 07 16:43:40.547144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:40.547101    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:40.547144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:40.547146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:40.559508 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:40.559474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:41.270986 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:41.270959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:43:47.216206 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:43:47.216131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:09.646105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:09.646076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g_01c751f9-ec80-4905-93a0-25e3b38a1c3a/payload-processing/0.log"
Sep 07 16:44:10.172861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.172828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:10.417890 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.417857    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"]
Sep 07 16:44:10.421837 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.421816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.424441 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.424388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1cc79d-openshift-default-dockercfg-5kq6b\""
Sep 07 16:44:10.431541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.431515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"]
Sep 07 16:44:10.443406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.443379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:10.551620 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-token\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551620 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7faa934-a6f7-403d-941a-baab66b12dbd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-credential-socket\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-envoy\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-certs\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlkk\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-kube-api-access-cwlkk\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-socket\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-podinfo\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.551906 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.551902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-data\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.643170 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.643133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:10.652386 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-envoy\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-certs\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlkk\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-kube-api-access-cwlkk\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-socket\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-podinfo\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652709 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-data\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652953 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-certs\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652953 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-token\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652953 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7faa934-a6f7-403d-941a-baab66b12dbd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.652953 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-socket\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.653128 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.652995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-credential-socket\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.653348 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.653325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-data\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.653527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.653353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-credential-socket\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.653643 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.653563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7faa934-a6f7-403d-941a-baab66b12dbd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.655084 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.655062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-envoy\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.655274 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.655259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-podinfo\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.659977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.659949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-token\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.660194 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.660176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlkk\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-kube-api-access-cwlkk\") pod \"e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") " pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:10.733060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:10.732972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:11.008401 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.008369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"]
Sep 07 16:44:11.010504 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:44:11.010470    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7faa934_a6f7_403d_941a_baab66b12dbd.slice/crio-6bae376034a160392d772b47ceba0e5dc00d88b12fe63ceed1e4ee9fce3d3391 WatchSource:0}: Error finding container 6bae376034a160392d772b47ceba0e5dc00d88b12fe63ceed1e4ee9fce3d3391: Status 404 returned error can't find the container with id 6bae376034a160392d772b47ceba0e5dc00d88b12fe63ceed1e4ee9fce3d3391
Sep 07 16:44:11.012995 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.012960    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:44:11.013096 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.013043    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:44:11.013096 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.013085    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:44:11.416653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.416621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" event={"ID":"a7faa934-a6f7-403d-941a-baab66b12dbd","Type":"ContainerStarted","Data":"5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"}
Sep 07 16:44:11.416861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.416674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" event={"ID":"a7faa934-a6f7-403d-941a-baab66b12dbd","Type":"ContainerStarted","Data":"6bae376034a160392d772b47ceba0e5dc00d88b12fe63ceed1e4ee9fce3d3391"}
Sep 07 16:44:11.438215 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.438162    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podStartSLOduration=1.438144602 podStartE2EDuration="1.438144602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:44:11.435828314 +0000 UTC m=+1456.317365279" watchObservedRunningTime="2026-09-07 16:44:11.438144602 +0000 UTC m=+1456.319681568"
Sep 07 16:44:11.733575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:11.733493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:12.739622 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:12.739591    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:13.298117 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:13.298072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:13.394711 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:13.394658    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:13.426660 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:13.426631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:13.427635 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:13.427616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:14.437539 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:14.437509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:14.490292 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:14.490264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:14.928647 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:14.928459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:15.006859 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:15.006824    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:32.527765 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:32.527729    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerID="5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160" exitCode=0
Sep 07 16:44:32.528171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:32.527793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" event={"ID":"a7faa934-a6f7-403d-941a-baab66b12dbd","Type":"ContainerDied","Data":"5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"}
Sep 07 16:44:32.528213 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:32.528202    2577 scope.go:117] "RemoveContainer" containerID="5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"
Sep 07 16:44:32.529399 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:32.529369    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:44:32.529462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:32.529438    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:44:32.529497 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:32.529469    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 07 16:44:33.539065 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:33.539032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" event={"ID":"a7faa934-a6f7-403d-941a-baab66b12dbd","Type":"ContainerStarted","Data":"97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a"}
Sep 07 16:44:33.733577 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:33.733531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:33.733817 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:33.733735    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 07 16:44:33.733817 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:33.733796    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 07 16:44:34.734388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:34.734348    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 07 16:44:34.734860 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:34.734421    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 07 16:44:35.734289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:35.734249    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 07 16:44:35.734481 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:35.734323    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 07 16:44:36.734420 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:36.734380    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 07 16:44:36.734600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:36.734452    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 07 16:44:37.733541 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:37.733508    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 07 16:44:37.733731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:37.733562    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 07 16:44:38.239272 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.239229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:38.734366 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.734324    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 07 16:44:38.734578 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.734429    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 07 16:44:38.890009 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.889974    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"]
Sep 07 16:44:38.893400 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.893380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:38.896733 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.896707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:44:38.899642 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:38.899615    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"]
Sep 07 16:44:39.047112 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.047023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9998b0ff-065d-438f-a8cd-09f7ec9c87e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h\" (UID: \"9998b0ff-065d-438f-a8cd-09f7ec9c87e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:39.147806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.147747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9998b0ff-065d-438f-a8cd-09f7ec9c87e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h\" (UID: \"9998b0ff-065d-438f-a8cd-09f7ec9c87e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:39.148335 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.148317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9998b0ff-065d-438f-a8cd-09f7ec9c87e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h\" (UID: \"9998b0ff-065d-438f-a8cd-09f7ec9c87e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:39.207692 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.207656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:39.475070 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.475044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"]
Sep 07 16:44:39.477925 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:44:39.477895    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9998b0ff_065d_438f_a8cd_09f7ec9c87e8.slice/crio-f406120406a7cca8ec382d38aae6ba7b46305ab3ddfd87a13d2a053cbb1af21a WatchSource:0}: Error finding container f406120406a7cca8ec382d38aae6ba7b46305ab3ddfd87a13d2a053cbb1af21a: Status 404 returned error can't find the container with id f406120406a7cca8ec382d38aae6ba7b46305ab3ddfd87a13d2a053cbb1af21a
Sep 07 16:44:39.572164 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.572131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h" event={"ID":"9998b0ff-065d-438f-a8cd-09f7ec9c87e8","Type":"ContainerStarted","Data":"f406120406a7cca8ec382d38aae6ba7b46305ab3ddfd87a13d2a053cbb1af21a"}
Sep 07 16:44:39.737697 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.737622    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:39.738045 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.738028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:39.738627 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:39.738612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:40.578710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:40.578618    2577 generic.go:356] "Generic (PLEG): container finished" podID="9998b0ff-065d-438f-a8cd-09f7ec9c87e8" containerID="d92639097ec978b86f7e053caabf5bbbfc30ac2345f4b061b3e52cb43ba31dd7" exitCode=0
Sep 07 16:44:40.578710 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:40.578691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h" event={"ID":"9998b0ff-065d-438f-a8cd-09f7ec9c87e8","Type":"ContainerDied","Data":"d92639097ec978b86f7e053caabf5bbbfc30ac2345f4b061b3e52cb43ba31dd7"}
Sep 07 16:44:41.711325 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:41.711299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:41.879020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:41.878925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9998b0ff-065d-438f-a8cd-09f7ec9c87e8-maas-api-service-ca\") pod \"9998b0ff-065d-438f-a8cd-09f7ec9c87e8\" (UID: \"9998b0ff-065d-438f-a8cd-09f7ec9c87e8\") "
Sep 07 16:44:41.879310 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:41.879287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9998b0ff-065d-438f-a8cd-09f7ec9c87e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9998b0ff-065d-438f-a8cd-09f7ec9c87e8" (UID: "9998b0ff-065d-438f-a8cd-09f7ec9c87e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:44:41.980041 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:41.980012    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9998b0ff-065d-438f-a8cd-09f7ec9c87e8-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:42.589597 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:42.589570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"
Sep 07 16:44:42.589597 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:42.589588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h" event={"ID":"9998b0ff-065d-438f-a8cd-09f7ec9c87e8","Type":"ContainerDied","Data":"f406120406a7cca8ec382d38aae6ba7b46305ab3ddfd87a13d2a053cbb1af21a"}
Sep 07 16:44:42.589829 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:42.589627    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f406120406a7cca8ec382d38aae6ba7b46305ab3ddfd87a13d2a053cbb1af21a"
Sep 07 16:44:44.897937 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:44.897900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:44.997834 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:44.997784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:45.074992 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.074895    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"]
Sep 07 16:44:45.077157 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.077131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:45.077615 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.077597    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1cc79d-mr67h"]
Sep 07 16:44:45.130314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.130275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:45.422037 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.421990    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:45.567600 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.567559    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9998b0ff-065d-438f-a8cd-09f7ec9c87e8" path="/var/lib/kubelet/pods/9998b0ff-065d-438f-a8cd-09f7ec9c87e8/volumes"
Sep 07 16:44:45.915408 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:45.915375    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:46.015613 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:46.015578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:50.992921 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:50.992889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:51.232120 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:51.232088    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"]
Sep 07 16:44:51.232374 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:51.232349    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" containerID="cri-o://97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a" gracePeriod=30
Sep 07 16:44:52.249733 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:52.249700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:54.740142 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:54.740103    2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 16:44:54.740540 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:54.740182    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 16:44:54.864438 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:54.864399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:44:56.489558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.489530    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:56.522327 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-credential-socket\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522327 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwlkk\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-kube-api-access-cwlkk\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-token\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-podinfo\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522439    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-envoy\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522471    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-socket\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-data\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522550 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522539    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-certs\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522880 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7faa934-a6f7-403d-941a-baab66b12dbd-istiod-ca-cert\") pod \"a7faa934-a6f7-403d-941a-baab66b12dbd\" (UID: \"a7faa934-a6f7-403d-941a-baab66b12dbd\") "
Sep 07 16:44:56.522880 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:44:56.522880 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:44:56.523019 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522944    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.523019 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.522963    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.523019 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.523004    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:44:56.523337 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.523116    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:44:56.523337 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.523171    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7faa934-a6f7-403d-941a-baab66b12dbd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:44:56.524824 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.524779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:44:56.524968 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.524868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-kube-api-access-cwlkk" (OuterVolumeSpecName: "kube-api-access-cwlkk") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "kube-api-access-cwlkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:44:56.525024 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.524970    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:44:56.525385 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.525364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7faa934-a6f7-403d-941a-baab66b12dbd" (UID: "a7faa934-a6f7-403d-941a-baab66b12dbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:44:56.624570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624522    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.624570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624561    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.624570 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624575    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7faa934-a6f7-403d-941a-baab66b12dbd-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.624894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624588    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7faa934-a6f7-403d-941a-baab66b12dbd-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.624894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624600    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlkk\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-kube-api-access-cwlkk\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.624894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624614    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.624894 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.624626    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7faa934-a6f7-403d-941a-baab66b12dbd-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:44:56.665011 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.664969    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerID="97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a" exitCode=0
Sep 07 16:44:56.665153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.665033    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"
Sep 07 16:44:56.665153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.665056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" event={"ID":"a7faa934-a6f7-403d-941a-baab66b12dbd","Type":"ContainerDied","Data":"97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a"}
Sep 07 16:44:56.665153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.665117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf" event={"ID":"a7faa934-a6f7-403d-941a-baab66b12dbd","Type":"ContainerDied","Data":"6bae376034a160392d772b47ceba0e5dc00d88b12fe63ceed1e4ee9fce3d3391"}
Sep 07 16:44:56.665153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.665142    2577 scope.go:117] "RemoveContainer" containerID="97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a"
Sep 07 16:44:56.675936 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.675912    2577 scope.go:117] "RemoveContainer" containerID="5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"
Sep 07 16:44:56.684678 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.684556    2577 scope.go:117] "RemoveContainer" containerID="97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a"
Sep 07 16:44:56.685265 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:44:56.685218    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a\": container with ID starting with 97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a not found: ID does not exist" containerID="97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a"
Sep 07 16:44:56.685372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.685260    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a"} err="failed to get container status \"97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a\": rpc error: code = NotFound desc = could not find container \"97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a\": container with ID starting with 97504267917a14a79d75b7304eca41619d6d9e46efe64a67e540eb43c8e12f2a not found: ID does not exist"
Sep 07 16:44:56.685372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.685286    2577 scope.go:117] "RemoveContainer" containerID="5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"
Sep 07 16:44:56.685663 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:44:56.685631    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160\": container with ID starting with 5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160 not found: ID does not exist" containerID="5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"
Sep 07 16:44:56.685735 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.685664    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160"} err="failed to get container status \"5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160\": rpc error: code = NotFound desc = could not find container \"5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160\": container with ID starting with 5f83c650fef4ce05f182bbfb5a9a1e6f382f5005ef777bebc32e48e570bf6160 not found: ID does not exist"
Sep 07 16:44:56.687675 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.687655    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"]
Sep 07 16:44:56.691661 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:56.691636    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cc79d-openshift-default-6b7f655d98-2bwsf"]
Sep 07 16:44:57.567975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:44:57.567944    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" path="/var/lib/kubelet/pods/a7faa934-a6f7-403d-941a-baab66b12dbd/volumes"
Sep 07 16:45:00.136253 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.135852    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"]
Sep 07 16:45:00.137171 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137147    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy"
Sep 07 16:45:00.137231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137175    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy"
Sep 07 16:45:00.137231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137214    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy"
Sep 07 16:45:00.137231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy"
Sep 07 16:45:00.137322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137242    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9998b0ff-065d-438f-a8cd-09f7ec9c87e8" containerName="revoke-keys"
Sep 07 16:45:00.137322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137250    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9998b0ff-065d-438f-a8cd-09f7ec9c87e8" containerName="revoke-keys"
Sep 07 16:45:00.137387 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137345    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy"
Sep 07 16:45:00.137387 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137357    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7faa934-a6f7-403d-941a-baab66b12dbd" containerName="istio-proxy"
Sep 07 16:45:00.137387 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.137366    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9998b0ff-065d-438f-a8cd-09f7ec9c87e8" containerName="revoke-keys"
Sep 07 16:45:00.142249 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.142227    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"]
Sep 07 16:45:00.142380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.142356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"
Sep 07 16:45:00.145101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.145075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:45:00.146050 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.146029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"
Sep 07 16:45:00.148690 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.148621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"]
Sep 07 16:45:00.152459 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.152425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"]
Sep 07 16:45:00.160421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.160399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"
Sep 07 16:45:00.169542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.169519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"
Sep 07 16:45:00.440528 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.440501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"]
Sep 07 16:45:00.442661 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:45:00.442632    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f05c00_bec5_4e16_a93f_33553f8c91e6.slice/crio-b12f3183bf36f8dba509d3f67b1c35142c42431fabd6c9190d00d3df84597dd0 WatchSource:0}: Error finding container b12f3183bf36f8dba509d3f67b1c35142c42431fabd6c9190d00d3df84597dd0: Status 404 returned error can't find the container with id b12f3183bf36f8dba509d3f67b1c35142c42431fabd6c9190d00d3df84597dd0
Sep 07 16:45:00.449153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.449128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"]
Sep 07 16:45:00.452488 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:45:00.452464    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8830d650_9b30_420d_b523_c408a6c297e7.slice/crio-08b5507431882b603733781545c04610433d83c8624c378ffed9df23560ea85b WatchSource:0}: Error finding container 08b5507431882b603733781545c04610433d83c8624c378ffed9df23560ea85b: Status 404 returned error can't find the container with id 08b5507431882b603733781545c04610433d83c8624c378ffed9df23560ea85b
Sep 07 16:45:00.697313 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.697211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght" event={"ID":"e3f05c00-bec5-4e16-a93f-33553f8c91e6","Type":"ContainerStarted","Data":"b12f3183bf36f8dba509d3f67b1c35142c42431fabd6c9190d00d3df84597dd0"}
Sep 07 16:45:00.698381 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:00.698357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn" event={"ID":"8830d650-9b30-420d-b523-c408a6c297e7","Type":"ContainerStarted","Data":"08b5507431882b603733781545c04610433d83c8624c378ffed9df23560ea85b"}
Sep 07 16:45:01.705308 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:01.705271    2577 generic.go:356] "Generic (PLEG): container finished" podID="e3f05c00-bec5-4e16-a93f-33553f8c91e6" containerID="b96f70d3f601c57483a344ef0a89083379e3cb16aa1466bc2450897e5ee6ddce" exitCode=0
Sep 07 16:45:01.705728 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:01.705317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght" event={"ID":"e3f05c00-bec5-4e16-a93f-33553f8c91e6","Type":"ContainerDied","Data":"b96f70d3f601c57483a344ef0a89083379e3cb16aa1466bc2450897e5ee6ddce"}
Sep 07 16:45:01.707048 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:01.707022    2577 generic.go:356] "Generic (PLEG): container finished" podID="8830d650-9b30-420d-b523-c408a6c297e7" containerID="603fbeb92d9d041059568819b0516990977e5ce1790bb7adfd0879bcf98524e9" exitCode=0
Sep 07 16:45:01.707163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:01.707098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn" event={"ID":"8830d650-9b30-420d-b523-c408a6c297e7","Type":"ContainerDied","Data":"603fbeb92d9d041059568819b0516990977e5ce1790bb7adfd0879bcf98524e9"}
Sep 07 16:45:02.867421 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:02.867399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"
Sep 07 16:45:02.870307 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:02.870279    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"
Sep 07 16:45:03.719281 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:03.719254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"
Sep 07 16:45:03.719475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:03.719282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght" event={"ID":"e3f05c00-bec5-4e16-a93f-33553f8c91e6","Type":"ContainerDied","Data":"b12f3183bf36f8dba509d3f67b1c35142c42431fabd6c9190d00d3df84597dd0"}
Sep 07 16:45:03.719475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:03.719333    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12f3183bf36f8dba509d3f67b1c35142c42431fabd6c9190d00d3df84597dd0"
Sep 07 16:45:03.720936 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:03.720906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn" event={"ID":"8830d650-9b30-420d-b523-c408a6c297e7","Type":"ContainerDied","Data":"08b5507431882b603733781545c04610433d83c8624c378ffed9df23560ea85b"}
Sep 07 16:45:03.720936 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:03.720934    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813325-t9tnn"
Sep 07 16:45:03.721141 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:03.720954    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b5507431882b603733781545c04610433d83c8624c378ffed9df23560ea85b"
Sep 07 16:45:10.270076 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.270036    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"]
Sep 07 16:45:10.271417 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.271390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:10.271948 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.271928    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4180e8-29813325-44ght"]
Sep 07 16:45:10.291032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.291007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"]
Sep 07 16:45:10.291225 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.291196    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g" podUID="01c751f9-ec80-4905-93a0-25e3b38a1c3a" containerName="payload-processing" containerID="cri-o://c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251" gracePeriod=30
Sep 07 16:45:10.370713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.370675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:10.527393 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.527283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:10.558726 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.558704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:45:10.580998 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.580969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:10.665148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.665119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ht2d\" (UniqueName: \"kubernetes.io/projected/01c751f9-ec80-4905-93a0-25e3b38a1c3a-kube-api-access-9ht2d\") pod \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") "
Sep 07 16:45:10.665334 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.665259    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c751f9-ec80-4905-93a0-25e3b38a1c3a-plugins-config-volume\") pod \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\" (UID: \"01c751f9-ec80-4905-93a0-25e3b38a1c3a\") "
Sep 07 16:45:10.665636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.665609    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c751f9-ec80-4905-93a0-25e3b38a1c3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01c751f9-ec80-4905-93a0-25e3b38a1c3a" (UID: "01c751f9-ec80-4905-93a0-25e3b38a1c3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:45:10.667422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.667396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c751f9-ec80-4905-93a0-25e3b38a1c3a-kube-api-access-9ht2d" (OuterVolumeSpecName: "kube-api-access-9ht2d") pod "01c751f9-ec80-4905-93a0-25e3b38a1c3a" (UID: "01c751f9-ec80-4905-93a0-25e3b38a1c3a"). InnerVolumeSpecName "kube-api-access-9ht2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:45:10.762528 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.762487    2577 generic.go:356] "Generic (PLEG): container finished" podID="01c751f9-ec80-4905-93a0-25e3b38a1c3a" containerID="c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251" exitCode=0
Sep 07 16:45:10.762729 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.762538    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"
Sep 07 16:45:10.762959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.762820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g" event={"ID":"01c751f9-ec80-4905-93a0-25e3b38a1c3a","Type":"ContainerDied","Data":"c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251"}
Sep 07 16:45:10.762959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.762896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g" event={"ID":"01c751f9-ec80-4905-93a0-25e3b38a1c3a","Type":"ContainerDied","Data":"d3433255e0fead20d3d0e5185b166faf53170f3d80deac70540e9aaaf4b48f36"}
Sep 07 16:45:10.762959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.762938    2577 scope.go:117] "RemoveContainer" containerID="c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251"
Sep 07 16:45:10.766226 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.766203    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c751f9-ec80-4905-93a0-25e3b38a1c3a-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:10.766324 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.766225    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ht2d\" (UniqueName: \"kubernetes.io/projected/01c751f9-ec80-4905-93a0-25e3b38a1c3a-kube-api-access-9ht2d\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:10.772477 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.772459    2577 scope.go:117] "RemoveContainer" containerID="c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251"
Sep 07 16:45:10.772796 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:45:10.772744    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251\": container with ID starting with c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251 not found: ID does not exist" containerID="c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251"
Sep 07 16:45:10.772896 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.772804    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251"} err="failed to get container status \"c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251\": rpc error: code = NotFound desc = could not find container \"c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251\": container with ID starting with c693be00c66461af66987b0ac75d329c5165face1a70b1e99a605d3a27841251 not found: ID does not exist"
Sep 07 16:45:10.783388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.783329    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"]
Sep 07 16:45:10.787022 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.787001    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4180e8-6d4cbf754f-84g5g"]
Sep 07 16:45:10.804662 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:10.804641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:11.571776 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:11.568968    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c751f9-ec80-4905-93a0-25e3b38a1c3a" path="/var/lib/kubelet/pods/01c751f9-ec80-4905-93a0-25e3b38a1c3a/volumes"
Sep 07 16:45:11.571776 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:11.569461    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f05c00-bec5-4e16-a93f-33553f8c91e6" path="/var/lib/kubelet/pods/e3f05c00-bec5-4e16-a93f-33553f8c91e6/volumes"
Sep 07 16:45:15.782328 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:15.782288    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"]
Sep 07 16:45:15.825411 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:15.825378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:30.802845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:30.802739    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69" podUID="f3062207-d164-4a7a-b7de-f4a4ad38c875" containerName="main" containerID="cri-o://038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8" gracePeriod=45
Sep 07 16:45:31.060204 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.060139    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:45:31.164470 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-tmp-dir\") pod \"f3062207-d164-4a7a-b7de-f4a4ad38c875\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") "
Sep 07 16:45:31.164665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-model-cache\") pod \"f3062207-d164-4a7a-b7de-f4a4ad38c875\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") "
Sep 07 16:45:31.164665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-dshm\") pod \"f3062207-d164-4a7a-b7de-f4a4ad38c875\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") "
Sep 07 16:45:31.164665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-home\") pod \"f3062207-d164-4a7a-b7de-f4a4ad38c875\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") "
Sep 07 16:45:31.164665 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3062207-d164-4a7a-b7de-f4a4ad38c875-tls-certs\") pod \"f3062207-d164-4a7a-b7de-f4a4ad38c875\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") "
Sep 07 16:45:31.164929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-splsm\" (UniqueName: \"kubernetes.io/projected/f3062207-d164-4a7a-b7de-f4a4ad38c875-kube-api-access-splsm\") pod \"f3062207-d164-4a7a-b7de-f4a4ad38c875\" (UID: \"f3062207-d164-4a7a-b7de-f4a4ad38c875\") "
Sep 07 16:45:31.164929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3062207-d164-4a7a-b7de-f4a4ad38c875" (UID: "f3062207-d164-4a7a-b7de-f4a4ad38c875"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:45:31.164929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164873    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3062207-d164-4a7a-b7de-f4a4ad38c875" (UID: "f3062207-d164-4a7a-b7de-f4a4ad38c875"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:45:31.164929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.164910    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-home" (OuterVolumeSpecName: "home") pod "f3062207-d164-4a7a-b7de-f4a4ad38c875" (UID: "f3062207-d164-4a7a-b7de-f4a4ad38c875"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:45:31.165105 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.165086    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:31.165144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.165109    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:31.165144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.165123    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:31.167062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.167029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3062207-d164-4a7a-b7de-f4a4ad38c875-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3062207-d164-4a7a-b7de-f4a4ad38c875" (UID: "f3062207-d164-4a7a-b7de-f4a4ad38c875"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:45:31.167062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.167028    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-dshm" (OuterVolumeSpecName: "dshm") pod "f3062207-d164-4a7a-b7de-f4a4ad38c875" (UID: "f3062207-d164-4a7a-b7de-f4a4ad38c875"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:45:31.167216 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.167095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3062207-d164-4a7a-b7de-f4a4ad38c875-kube-api-access-splsm" (OuterVolumeSpecName: "kube-api-access-splsm") pod "f3062207-d164-4a7a-b7de-f4a4ad38c875" (UID: "f3062207-d164-4a7a-b7de-f4a4ad38c875"). InnerVolumeSpecName "kube-api-access-splsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:45:31.265825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.265787    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3062207-d164-4a7a-b7de-f4a4ad38c875-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:31.265825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.265815    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3062207-d164-4a7a-b7de-f4a4ad38c875-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:31.265825 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.265827    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splsm\" (UniqueName: \"kubernetes.io/projected/f3062207-d164-4a7a-b7de-f4a4ad38c875-kube-api-access-splsm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:31.872135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.872046    2577 generic.go:356] "Generic (PLEG): container finished" podID="f3062207-d164-4a7a-b7de-f4a4ad38c875" containerID="038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8" exitCode=0
Sep 07 16:45:31.872135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.872101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"
Sep 07 16:45:31.872658 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.872134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69" event={"ID":"f3062207-d164-4a7a-b7de-f4a4ad38c875","Type":"ContainerDied","Data":"038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8"}
Sep 07 16:45:31.872658 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.872184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69" event={"ID":"f3062207-d164-4a7a-b7de-f4a4ad38c875","Type":"ContainerDied","Data":"b7a7972a4b01334d88d5474a81238141897e0a8d934c0640c36c22502fa7f8ec"}
Sep 07 16:45:31.872658 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.872209    2577 scope.go:117] "RemoveContainer" containerID="038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8"
Sep 07 16:45:31.881002 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.880962    2577 scope.go:117] "RemoveContainer" containerID="038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8"
Sep 07 16:45:31.881263 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:45:31.881246    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8\": container with ID starting with 038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8 not found: ID does not exist" containerID="038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8"
Sep 07 16:45:31.881327 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.881275    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8"} err="failed to get container status \"038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8\": rpc error: code = NotFound desc = could not find container \"038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8\": container with ID starting with 038041936fe2b7c0497eda0ed365f3536fb7a386d103a774bb3d290fb05902d8 not found: ID does not exist"
Sep 07 16:45:31.888243 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.888223    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"]
Sep 07 16:45:31.890901 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:31.890882    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4180e8/ipp-route-4180e8-kserve-6bbfdbb6f9-92l69"]
Sep 07 16:45:33.568098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:33.568067    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3062207-d164-4a7a-b7de-f4a4ad38c875" path="/var/lib/kubelet/pods/f3062207-d164-4a7a-b7de-f4a4ad38c875/volumes"
Sep 07 16:45:39.560069 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:39.560039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:40.397000 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:40.396972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:46.005402 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.005373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:46.105491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.105465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:46.277478 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.277386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:46.625180 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.625094    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"]
Sep 07 16:45:46.626001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.625978    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c751f9-ec80-4905-93a0-25e3b38a1c3a" containerName="payload-processing"
Sep 07 16:45:46.626148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626137    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c751f9-ec80-4905-93a0-25e3b38a1c3a" containerName="payload-processing"
Sep 07 16:45:46.626233 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8830d650-9b30-420d-b523-c408a6c297e7" containerName="cleanup"
Sep 07 16:45:46.626299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626292    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8830d650-9b30-420d-b523-c408a6c297e7" containerName="cleanup"
Sep 07 16:45:46.626380 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626372    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f05c00-bec5-4e16-a93f-33553f8c91e6" containerName="cleanup"
Sep 07 16:45:46.626445 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626438    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f05c00-bec5-4e16-a93f-33553f8c91e6" containerName="cleanup"
Sep 07 16:45:46.626533 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626525    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3062207-d164-4a7a-b7de-f4a4ad38c875" containerName="main"
Sep 07 16:45:46.626597 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626589    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3062207-d164-4a7a-b7de-f4a4ad38c875" containerName="main"
Sep 07 16:45:46.626813 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626802    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c751f9-ec80-4905-93a0-25e3b38a1c3a" containerName="payload-processing"
Sep 07 16:45:46.626914 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626905    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8830d650-9b30-420d-b523-c408a6c297e7" containerName="cleanup"
Sep 07 16:45:46.626986 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.626978    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3062207-d164-4a7a-b7de-f4a4ad38c875" containerName="main"
Sep 07 16:45:46.627055 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.627048    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f05c00-bec5-4e16-a93f-33553f8c91e6" containerName="cleanup"
Sep 07 16:45:46.632097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.632075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:46.634797 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.634778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:45:46.635862 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.635838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"]
Sep 07 16:45:46.719704 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.719663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835d11c-3a19-4add-afd2-0efb0d021aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf\" (UID: \"c835d11c-3a19-4add-afd2-0efb0d021aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:46.761071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.761031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:46.820761 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.820724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835d11c-3a19-4add-afd2-0efb0d021aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf\" (UID: \"c835d11c-3a19-4add-afd2-0efb0d021aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:46.821321 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.821303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835d11c-3a19-4add-afd2-0efb0d021aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf\" (UID: \"c835d11c-3a19-4add-afd2-0efb0d021aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:46.944503 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:46.944460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:47.221881 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:47.221857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"]
Sep 07 16:45:47.223830 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:45:47.223804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc835d11c_3a19_4add_afd2_0efb0d021aeb.slice/crio-d4caf290595ad5ed60bf529c5c3d63504637bbb3b3880349e33d0f6e43d0e1c7 WatchSource:0}: Error finding container d4caf290595ad5ed60bf529c5c3d63504637bbb3b3880349e33d0f6e43d0e1c7: Status 404 returned error can't find the container with id d4caf290595ad5ed60bf529c5c3d63504637bbb3b3880349e33d0f6e43d0e1c7
Sep 07 16:45:47.961019 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:47.960989    2577 generic.go:356] "Generic (PLEG): container finished" podID="c835d11c-3a19-4add-afd2-0efb0d021aeb" containerID="8c90d931a4c892b3b8733002dae74e3735940ad406c436b9ef5ffdc526ee74a6" exitCode=0
Sep 07 16:45:47.961188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:47.961086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf" event={"ID":"c835d11c-3a19-4add-afd2-0efb0d021aeb","Type":"ContainerDied","Data":"8c90d931a4c892b3b8733002dae74e3735940ad406c436b9ef5ffdc526ee74a6"}
Sep 07 16:45:47.961188 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:47.961124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf" event={"ID":"c835d11c-3a19-4add-afd2-0efb0d021aeb","Type":"ContainerStarted","Data":"d4caf290595ad5ed60bf529c5c3d63504637bbb3b3880349e33d0f6e43d0e1c7"}
Sep 07 16:45:49.097463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.097435    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:49.245526 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.245429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835d11c-3a19-4add-afd2-0efb0d021aeb-maas-api-service-ca\") pod \"c835d11c-3a19-4add-afd2-0efb0d021aeb\" (UID: \"c835d11c-3a19-4add-afd2-0efb0d021aeb\") "
Sep 07 16:45:49.245824 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.245801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c835d11c-3a19-4add-afd2-0efb0d021aeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c835d11c-3a19-4add-afd2-0efb0d021aeb" (UID: "c835d11c-3a19-4add-afd2-0efb0d021aeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:45:49.346580 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.346546    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835d11c-3a19-4add-afd2-0efb0d021aeb-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:49.972442 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.972411    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"
Sep 07 16:45:49.972653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.972452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf" event={"ID":"c835d11c-3a19-4add-afd2-0efb0d021aeb","Type":"ContainerDied","Data":"d4caf290595ad5ed60bf529c5c3d63504637bbb3b3880349e33d0f6e43d0e1c7"}
Sep 07 16:45:49.972653 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:49.972501    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4caf290595ad5ed60bf529c5c3d63504637bbb3b3880349e33d0f6e43d0e1c7"
Sep 07 16:45:57.076144 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.076111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:57.108851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.108805    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"]
Sep 07 16:45:57.109103 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.109077    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr" podUID="d09d3c6d-09aa-488b-afca-3107a68afc8b" containerName="payload-pre-processing" containerID="cri-o://676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b" gracePeriod=30
Sep 07 16:45:57.174638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.174601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:57.264196 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.264156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"]
Sep 07 16:45:57.266840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.266809    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a45e8-k2drf"]
Sep 07 16:45:57.266955 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.266922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:57.335555 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.335474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:57.376549 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.376525    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:45:57.523249 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.523215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsq5t\" (UniqueName: \"kubernetes.io/projected/d09d3c6d-09aa-488b-afca-3107a68afc8b-kube-api-access-zsq5t\") pod \"d09d3c6d-09aa-488b-afca-3107a68afc8b\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") "
Sep 07 16:45:57.523456 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.523336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09d3c6d-09aa-488b-afca-3107a68afc8b-plugins-config-volume\") pod \"d09d3c6d-09aa-488b-afca-3107a68afc8b\" (UID: \"d09d3c6d-09aa-488b-afca-3107a68afc8b\") "
Sep 07 16:45:57.523739 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.523717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09d3c6d-09aa-488b-afca-3107a68afc8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d09d3c6d-09aa-488b-afca-3107a68afc8b" (UID: "d09d3c6d-09aa-488b-afca-3107a68afc8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:45:57.525518 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.525495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09d3c6d-09aa-488b-afca-3107a68afc8b-kube-api-access-zsq5t" (OuterVolumeSpecName: "kube-api-access-zsq5t") pod "d09d3c6d-09aa-488b-afca-3107a68afc8b" (UID: "d09d3c6d-09aa-488b-afca-3107a68afc8b"). InnerVolumeSpecName "kube-api-access-zsq5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:45:57.568056 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.568025    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c835d11c-3a19-4add-afd2-0efb0d021aeb" path="/var/lib/kubelet/pods/c835d11c-3a19-4add-afd2-0efb0d021aeb/volumes"
Sep 07 16:45:57.625217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.625137    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09d3c6d-09aa-488b-afca-3107a68afc8b-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:57.625217 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.625165    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsq5t\" (UniqueName: \"kubernetes.io/projected/d09d3c6d-09aa-488b-afca-3107a68afc8b-kube-api-access-zsq5t\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:45:57.636929 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:57.636904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:45:58.016677 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.016649    2577 generic.go:356] "Generic (PLEG): container finished" podID="d09d3c6d-09aa-488b-afca-3107a68afc8b" containerID="676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b" exitCode=0
Sep 07 16:45:58.016842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.016689    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"
Sep 07 16:45:58.016842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.016730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr" event={"ID":"d09d3c6d-09aa-488b-afca-3107a68afc8b","Type":"ContainerDied","Data":"676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b"}
Sep 07 16:45:58.016842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.016799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr" event={"ID":"d09d3c6d-09aa-488b-afca-3107a68afc8b","Type":"ContainerDied","Data":"dee7ff8e5622b970b5254cf6eb681eda92b0123ede6555032ddd8cb2d2b10e72"}
Sep 07 16:45:58.016842 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.016823    2577 scope.go:117] "RemoveContainer" containerID="676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b"
Sep 07 16:45:58.025543 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.025526    2577 scope.go:117] "RemoveContainer" containerID="676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b"
Sep 07 16:45:58.025822 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:45:58.025802    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b\": container with ID starting with 676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b not found: ID does not exist" containerID="676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b"
Sep 07 16:45:58.025885 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.025831    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b"} err="failed to get container status \"676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b\": rpc error: code = NotFound desc = could not find container \"676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b\": container with ID starting with 676f612cf899be6dc6c0760531b8164a932bad81e7e06f638bd70d0b7bb67f0b not found: ID does not exist"
Sep 07 16:45:58.034696 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.034678    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"]
Sep 07 16:45:58.041009 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:58.040990    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a45e8-7cbd64b98-hhhkr"]
Sep 07 16:45:59.568098 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:45:59.568064    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d09d3c6d-09aa-488b-afca-3107a68afc8b" path="/var/lib/kubelet/pods/d09d3c6d-09aa-488b-afca-3107a68afc8b/volumes"
Sep 07 16:46:03.467473 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:03.467437    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:03.536900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:03.536866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:03.702884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:03.702854    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"]
Sep 07 16:46:03.703232 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:03.703081    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" containerID="cri-o://664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15" gracePeriod=30
Sep 07 16:46:03.724189 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:03.724101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:04.147447 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:04.147408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:05.895178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:05.895141    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 16:46:05.895563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:05.895216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 16:46:08.961388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:08.961362    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:46:09.028329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-token\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7kp2\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-kube-api-access-g7kp2\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-envoy\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028459    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-credential-socket\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-socket\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-certs\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-data\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istiod-ca-cert\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-podinfo\") pod \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\" (UID: \"5e2e6379-9c61-4533-affd-b9e3e28f9e4d\") "
Sep 07 16:46:09.028884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:09.028884 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:09.029049 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.028913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:09.029049 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.029009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:46:09.029165 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.029150    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.029223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.029166    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.029223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.029179    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.029223 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.029189    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.029479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.029356    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:09.030851 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.030830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:09.030922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.030874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:46:09.031089 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.031069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-kube-api-access-g7kp2" (OuterVolumeSpecName: "kube-api-access-g7kp2") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "kube-api-access-g7kp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:09.031147 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.031081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e2e6379-9c61-4533-affd-b9e3e28f9e4d" (UID: "5e2e6379-9c61-4533-affd-b9e3e28f9e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:09.078882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.078844    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerID="664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15" exitCode=0
Sep 07 16:46:09.079032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.078894    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"
Sep 07 16:46:09.079032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.078922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" event={"ID":"5e2e6379-9c61-4533-affd-b9e3e28f9e4d","Type":"ContainerDied","Data":"664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15"}
Sep 07 16:46:09.079032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.078977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8" event={"ID":"5e2e6379-9c61-4533-affd-b9e3e28f9e4d","Type":"ContainerDied","Data":"a73958df0c61ebf7eb173231f2dfe02896e56d8087785dd5ce21eb0dddf561ce"}
Sep 07 16:46:09.079032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.079001    2577 scope.go:117] "RemoveContainer" containerID="664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15"
Sep 07 16:46:09.088222 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.088204    2577 scope.go:117] "RemoveContainer" containerID="5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"
Sep 07 16:46:09.096202 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.096186    2577 scope.go:117] "RemoveContainer" containerID="664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15"
Sep 07 16:46:09.096459 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:09.096441    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15\": container with ID starting with 664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15 not found: ID does not exist" containerID="664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15"
Sep 07 16:46:09.096505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.096468    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15"} err="failed to get container status \"664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15\": rpc error: code = NotFound desc = could not find container \"664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15\": container with ID starting with 664f9049559fb9b1795a97e4a21afc78c9caf19823e869151efc325b9d190c15 not found: ID does not exist"
Sep 07 16:46:09.096505 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.096483    2577 scope.go:117] "RemoveContainer" containerID="5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"
Sep 07 16:46:09.096730 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:09.096711    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6\": container with ID starting with 5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6 not found: ID does not exist" containerID="5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"
Sep 07 16:46:09.096794 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.096737    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6"} err="failed to get container status \"5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6\": rpc error: code = NotFound desc = could not find container \"5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6\": container with ID starting with 5797a1f480debf65b41f0cfa1e814571074e5dd2676c73d7da40f0eabab893f6 not found: ID does not exist"
Sep 07 16:46:09.100450 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.100431    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"]
Sep 07 16:46:09.104508 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.104490    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a45e8-openshift-default-6f48c7744b-w7bd8"]
Sep 07 16:46:09.130148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.130127    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.130148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.130149    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.130299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.130158    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7kp2\" (UniqueName: \"kubernetes.io/projected/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-kube-api-access-g7kp2\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.130299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.130166    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.130299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.130176    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2e6379-9c61-4533-affd-b9e3e28f9e4d-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:09.567179 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:09.567146    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" path="/var/lib/kubelet/pods/5e2e6379-9c61-4533-affd-b9e3e28f9e4d/volumes"
Sep 07 16:46:14.708638 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.708597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:14.732845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.732806    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"]
Sep 07 16:46:14.733148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.733078    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc" podUID="fc3123c5-96fe-432b-b90e-7b1e925250c7" containerName="payload-processing" containerID="cri-o://29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab" gracePeriod=30
Sep 07 16:46:14.741492 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.741460    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"]
Sep 07 16:46:14.742062 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.741684    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt" podUID="dd546456-a651-4f22-8939-f1b182e6f527" containerName="payload-pre-processing" containerID="cri-o://f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e" gracePeriod=30
Sep 07 16:46:14.793079 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.793043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:14.896060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.896024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:14.970793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:14.970681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:15.020192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.020168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:46:15.022866 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.022847    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:46:15.093577 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.093545    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd546456-a651-4f22-8939-f1b182e6f527-plugins-config-volume\") pod \"dd546456-a651-4f22-8939-f1b182e6f527\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") "
Sep 07 16:46:15.093840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.093590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3123c5-96fe-432b-b90e-7b1e925250c7-plugins-config-volume\") pod \"fc3123c5-96fe-432b-b90e-7b1e925250c7\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") "
Sep 07 16:46:15.093840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.093689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdtlb\" (UniqueName: \"kubernetes.io/projected/fc3123c5-96fe-432b-b90e-7b1e925250c7-kube-api-access-kdtlb\") pod \"fc3123c5-96fe-432b-b90e-7b1e925250c7\" (UID: \"fc3123c5-96fe-432b-b90e-7b1e925250c7\") "
Sep 07 16:46:15.093840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.093727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scncs\" (UniqueName: \"kubernetes.io/projected/dd546456-a651-4f22-8939-f1b182e6f527-kube-api-access-scncs\") pod \"dd546456-a651-4f22-8939-f1b182e6f527\" (UID: \"dd546456-a651-4f22-8939-f1b182e6f527\") "
Sep 07 16:46:15.094043 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.093977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd546456-a651-4f22-8939-f1b182e6f527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd546456-a651-4f22-8939-f1b182e6f527" (UID: "dd546456-a651-4f22-8939-f1b182e6f527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:46:15.094093 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.094042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3123c5-96fe-432b-b90e-7b1e925250c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc3123c5-96fe-432b-b90e-7b1e925250c7" (UID: "fc3123c5-96fe-432b-b90e-7b1e925250c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:46:15.096007 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.095981    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd546456-a651-4f22-8939-f1b182e6f527-kube-api-access-scncs" (OuterVolumeSpecName: "kube-api-access-scncs") pod "dd546456-a651-4f22-8939-f1b182e6f527" (UID: "dd546456-a651-4f22-8939-f1b182e6f527"). InnerVolumeSpecName "kube-api-access-scncs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:15.096108 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.096009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3123c5-96fe-432b-b90e-7b1e925250c7-kube-api-access-kdtlb" (OuterVolumeSpecName: "kube-api-access-kdtlb") pod "fc3123c5-96fe-432b-b90e-7b1e925250c7" (UID: "fc3123c5-96fe-432b-b90e-7b1e925250c7"). InnerVolumeSpecName "kube-api-access-kdtlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:15.114417 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.114387    2577 generic.go:356] "Generic (PLEG): container finished" podID="fc3123c5-96fe-432b-b90e-7b1e925250c7" containerID="29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab" exitCode=0
Sep 07 16:46:15.114592 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.114433    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"
Sep 07 16:46:15.114592 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.114462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc" event={"ID":"fc3123c5-96fe-432b-b90e-7b1e925250c7","Type":"ContainerDied","Data":"29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab"}
Sep 07 16:46:15.114592 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.114508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc" event={"ID":"fc3123c5-96fe-432b-b90e-7b1e925250c7","Type":"ContainerDied","Data":"90e9bd32cdd5dbf8df1d13fe61bc1d0315e2bc6e1e9243757d02d46ea5cba2eb"}
Sep 07 16:46:15.114592 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.114533    2577 scope.go:117] "RemoveContainer" containerID="29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab"
Sep 07 16:46:15.115937 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.115911    2577 generic.go:356] "Generic (PLEG): container finished" podID="dd546456-a651-4f22-8939-f1b182e6f527" containerID="f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e" exitCode=0
Sep 07 16:46:15.116046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.115952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"
Sep 07 16:46:15.116046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.115989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt" event={"ID":"dd546456-a651-4f22-8939-f1b182e6f527","Type":"ContainerDied","Data":"f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e"}
Sep 07 16:46:15.116046 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.116042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt" event={"ID":"dd546456-a651-4f22-8939-f1b182e6f527","Type":"ContainerDied","Data":"5a9f20c270b6b8fdffc06e3b33d915f747034da4b00a8dcc3807abe9068fa3e5"}
Sep 07 16:46:15.123860 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.123848    2577 scope.go:117] "RemoveContainer" containerID="29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab"
Sep 07 16:46:15.124136 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:15.124115    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab\": container with ID starting with 29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab not found: ID does not exist" containerID="29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab"
Sep 07 16:46:15.124230 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.124140    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab"} err="failed to get container status \"29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab\": rpc error: code = NotFound desc = could not find container \"29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab\": container with ID starting with 29741c17f9f001283eaf6c593b605fef78146dfac07bdbb826a618e1eb18d7ab not found: ID does not exist"
Sep 07 16:46:15.124230 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.124155    2577 scope.go:117] "RemoveContainer" containerID="f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e"
Sep 07 16:46:15.132156 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.132138    2577 scope.go:117] "RemoveContainer" containerID="f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e"
Sep 07 16:46:15.132468 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:15.132450    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e\": container with ID starting with f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e not found: ID does not exist" containerID="f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e"
Sep 07 16:46:15.132527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.132477    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e"} err="failed to get container status \"f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e\": rpc error: code = NotFound desc = could not find container \"f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e\": container with ID starting with f092cc2c2d5d9084139c58e27b8ad96c9e671a3f301897a6e5f772f00323c07e not found: ID does not exist"
Sep 07 16:46:15.138843 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.138819    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"]
Sep 07 16:46:15.142619 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.142599    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8d59e2-cf6cbdf66-p5xqt"]
Sep 07 16:46:15.153016 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.152980    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"]
Sep 07 16:46:15.156922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.156897    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8d59e2-69b5655485-bfkgc"]
Sep 07 16:46:15.195409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.195374    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdtlb\" (UniqueName: \"kubernetes.io/projected/fc3123c5-96fe-432b-b90e-7b1e925250c7-kube-api-access-kdtlb\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:15.195409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.195405    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scncs\" (UniqueName: \"kubernetes.io/projected/dd546456-a651-4f22-8939-f1b182e6f527-kube-api-access-scncs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:15.195574 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.195421    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd546456-a651-4f22-8939-f1b182e6f527-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:15.195574 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.195433    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3123c5-96fe-432b-b90e-7b1e925250c7-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:15.265538 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.265447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:15.567975 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.567888    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd546456-a651-4f22-8939-f1b182e6f527" path="/var/lib/kubelet/pods/dd546456-a651-4f22-8939-f1b182e6f527/volumes"
Sep 07 16:46:15.568262 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:15.568250    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3123c5-96fe-432b-b90e-7b1e925250c7" path="/var/lib/kubelet/pods/fc3123c5-96fe-432b-b90e-7b1e925250c7/volumes"
Sep 07 16:46:21.192636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.192600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:21.300803 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.300747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:21.423017 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.422986    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"]
Sep 07 16:46:21.423231 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.423204    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy" containerID="cri-o://5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af" gracePeriod=30
Sep 07 16:46:21.450667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.450588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:21.846276 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846198    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"]
Sep 07 16:46:21.846580 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846568    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd546456-a651-4f22-8939-f1b182e6f527" containerName="payload-pre-processing"
Sep 07 16:46:21.846621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846582    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd546456-a651-4f22-8939-f1b182e6f527" containerName="payload-pre-processing"
Sep 07 16:46:21.846621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846592    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy"
Sep 07 16:46:21.846621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846597    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy"
Sep 07 16:46:21.846621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846611    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09d3c6d-09aa-488b-afca-3107a68afc8b" containerName="payload-pre-processing"
Sep 07 16:46:21.846621 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846617    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09d3c6d-09aa-488b-afca-3107a68afc8b" containerName="payload-pre-processing"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846626    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3123c5-96fe-432b-b90e-7b1e925250c7" containerName="payload-processing"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846632    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3123c5-96fe-432b-b90e-7b1e925250c7" containerName="payload-processing"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c835d11c-3a19-4add-afd2-0efb0d021aeb" containerName="revoke-keys"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846643    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c835d11c-3a19-4add-afd2-0efb0d021aeb" containerName="revoke-keys"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846650    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846655    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846713    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846720    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09d3c6d-09aa-488b-afca-3107a68afc8b" containerName="payload-pre-processing"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846729    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3123c5-96fe-432b-b90e-7b1e925250c7" containerName="payload-processing"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846734    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c835d11c-3a19-4add-afd2-0efb0d021aeb" containerName="revoke-keys"
Sep 07 16:46:21.846799 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.846742    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd546456-a651-4f22-8939-f1b182e6f527" containerName="payload-pre-processing"
Sep 07 16:46:21.851640 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.851617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:21.854280 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.854260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\""
Sep 07 16:46:21.857624 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.857591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"]
Sep 07 16:46:21.887623 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.887594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:21.960890 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:21.960853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc661a47-b55b-405f-8900-68f55350303f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks\" (UID: \"fc661a47-b55b-405f-8900-68f55350303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:22.062000 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:22.061965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc661a47-b55b-405f-8900-68f55350303f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks\" (UID: \"fc661a47-b55b-405f-8900-68f55350303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:22.062522 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:22.062504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc661a47-b55b-405f-8900-68f55350303f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks\" (UID: \"fc661a47-b55b-405f-8900-68f55350303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:22.162780 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:22.162720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:22.432677 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:22.432644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"]
Sep 07 16:46:22.433804 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:46:22.433779    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc661a47_b55b_405f_8900_68f55350303f.slice/crio-f3a6ccea0a30e7259619d387a67624a7f959a216cfed172f19e31062d7c6e86e WatchSource:0}: Error finding container f3a6ccea0a30e7259619d387a67624a7f959a216cfed172f19e31062d7c6e86e: Status 404 returned error can't find the container with id f3a6ccea0a30e7259619d387a67624a7f959a216cfed172f19e31062d7c6e86e
Sep 07 16:46:23.160423 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:23.160380    2577 generic.go:356] "Generic (PLEG): container finished" podID="fc661a47-b55b-405f-8900-68f55350303f" containerID="9dc83fe165c074939f171b2671e411d28a064d8cf517818a56c81175e8e40e1d" exitCode=0
Sep 07 16:46:23.160625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:23.160455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks" event={"ID":"fc661a47-b55b-405f-8900-68f55350303f","Type":"ContainerDied","Data":"9dc83fe165c074939f171b2671e411d28a064d8cf517818a56c81175e8e40e1d"}
Sep 07 16:46:23.160625 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:23.160496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks" event={"ID":"fc661a47-b55b-405f-8900-68f55350303f","Type":"ContainerStarted","Data":"f3a6ccea0a30e7259619d387a67624a7f959a216cfed172f19e31062d7c6e86e"}
Sep 07 16:46:24.289083 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:24.289058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:24.383394 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:24.383361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc661a47-b55b-405f-8900-68f55350303f-maas-api-service-ca\") pod \"fc661a47-b55b-405f-8900-68f55350303f\" (UID: \"fc661a47-b55b-405f-8900-68f55350303f\") "
Sep 07 16:46:24.383793 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:24.383734    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc661a47-b55b-405f-8900-68f55350303f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc661a47-b55b-405f-8900-68f55350303f" (UID: "fc661a47-b55b-405f-8900-68f55350303f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:46:24.484867 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:24.484784    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc661a47-b55b-405f-8900-68f55350303f-maas-api-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:25.171832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:25.171804    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"
Sep 07 16:46:25.171832 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:25.171815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks" event={"ID":"fc661a47-b55b-405f-8900-68f55350303f","Type":"ContainerDied","Data":"f3a6ccea0a30e7259619d387a67624a7f959a216cfed172f19e31062d7c6e86e"}
Sep 07 16:46:25.172060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:25.171908    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3a6ccea0a30e7259619d387a67624a7f959a216cfed172f19e31062d7c6e86e"
Sep 07 16:46:26.676388 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.676363    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:46:26.807364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-data\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807307    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-envoy\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-certs\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xf75\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-kube-api-access-6xf75\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bffe79-62d9-4cbf-864d-fbe1067631eb-istiod-ca-cert\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-credential-socket\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-token\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-socket\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-podinfo\") pod \"89bffe79-62d9-4cbf-864d-fbe1067631eb\" (UID: \"89bffe79-62d9-4cbf-864d-fbe1067631eb\") "
Sep 07 16:46:26.807618 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:26.808072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:26.808072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:26.808072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bffe79-62d9-4cbf-864d-fbe1067631eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:46:26.808072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:26.808072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.807976    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.809864 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.809833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-kube-api-access-6xf75" (OuterVolumeSpecName: "kube-api-access-6xf75") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "kube-api-access-6xf75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:26.809982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.809885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:46:26.809982 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.809892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:26.810174 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.810154    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89bffe79-62d9-4cbf-864d-fbe1067631eb" (UID: "89bffe79-62d9-4cbf-864d-fbe1067631eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 16:46:26.909163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909130    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xf75\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-kube-api-access-6xf75\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909159    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bffe79-62d9-4cbf-864d-fbe1067631eb-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909168    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909177    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909186    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909194    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909202    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:26.909403 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:26.909209    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bffe79-62d9-4cbf-864d-fbe1067631eb-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:27.183828 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.183795    2577 generic.go:356] "Generic (PLEG): container finished" podID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerID="5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af" exitCode=0
Sep 07 16:46:27.184004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.183840    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"
Sep 07 16:46:27.184004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.183877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" event={"ID":"89bffe79-62d9-4cbf-864d-fbe1067631eb","Type":"ContainerDied","Data":"5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af"}
Sep 07 16:46:27.184004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.183929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt" event={"ID":"89bffe79-62d9-4cbf-864d-fbe1067631eb","Type":"ContainerDied","Data":"a8d6731457e5784d52a7e5127796046324ea24e0a897142bf8b624ccd895c8a6"}
Sep 07 16:46:27.184004 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.183952    2577 scope.go:117] "RemoveContainer" containerID="5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af"
Sep 07 16:46:27.193882 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.193867    2577 scope.go:117] "RemoveContainer" containerID="dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"
Sep 07 16:46:27.201480 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.201464    2577 scope.go:117] "RemoveContainer" containerID="5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af"
Sep 07 16:46:27.201737 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:27.201719    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af\": container with ID starting with 5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af not found: ID does not exist" containerID="5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af"
Sep 07 16:46:27.201801 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.201746    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af"} err="failed to get container status \"5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af\": rpc error: code = NotFound desc = could not find container \"5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af\": container with ID starting with 5af106dd9cd917a29ffb0d4ab651e72cf57be2085b7446cd17715b59623186af not found: ID does not exist"
Sep 07 16:46:27.201801 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.201779    2577 scope.go:117] "RemoveContainer" containerID="dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"
Sep 07 16:46:27.202019 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:27.202001    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470\": container with ID starting with dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470 not found: ID does not exist" containerID="dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"
Sep 07 16:46:27.202060 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.202025    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470"} err="failed to get container status \"dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470\": rpc error: code = NotFound desc = could not find container \"dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470\": container with ID starting with dbc7e697a3067349414b78d64dd65b163e20c1c2abf9559a0e53e99b66829470 not found: ID does not exist"
Sep 07 16:46:27.205501 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.205483    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"]
Sep 07 16:46:27.208670 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.208650    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8d59e2-openshift-default-c5c756f98-6hhgt"]
Sep 07 16:46:27.568275 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:27.568190    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" path="/var/lib/kubelet/pods/89bffe79-62d9-4cbf-864d-fbe1067631eb/volumes"
Sep 07 16:46:32.295159 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.295128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:32.321730 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.321693    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"]
Sep 07 16:46:32.321956 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.321924    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj" podUID="cb01fa9f-694f-423c-8d29-36832a892c8a" containerName="payload-processing" containerID="cri-o://36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85" gracePeriod=30
Sep 07 16:46:32.393559 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.393528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:32.485059 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.485018    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"]
Sep 07 16:46:32.487861 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.487837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:32.488125 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.488106    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7aca0f-hsrks"]
Sep 07 16:46:32.552024 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.551910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:32.573997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.573970    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:46:32.662964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.662936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb01fa9f-694f-423c-8d29-36832a892c8a-plugins-config-volume\") pod \"cb01fa9f-694f-423c-8d29-36832a892c8a\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") "
Sep 07 16:46:32.663150 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.663013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnz8l\" (UniqueName: \"kubernetes.io/projected/cb01fa9f-694f-423c-8d29-36832a892c8a-kube-api-access-hnz8l\") pod \"cb01fa9f-694f-423c-8d29-36832a892c8a\" (UID: \"cb01fa9f-694f-423c-8d29-36832a892c8a\") "
Sep 07 16:46:32.663364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.663340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb01fa9f-694f-423c-8d29-36832a892c8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb01fa9f-694f-423c-8d29-36832a892c8a" (UID: "cb01fa9f-694f-423c-8d29-36832a892c8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:46:32.665148 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.665125    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb01fa9f-694f-423c-8d29-36832a892c8a-kube-api-access-hnz8l" (OuterVolumeSpecName: "kube-api-access-hnz8l") pod "cb01fa9f-694f-423c-8d29-36832a892c8a" (UID: "cb01fa9f-694f-423c-8d29-36832a892c8a"). InnerVolumeSpecName "kube-api-access-hnz8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:46:32.764135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.764100    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb01fa9f-694f-423c-8d29-36832a892c8a-plugins-config-volume\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:32.764135 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.764127    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnz8l\" (UniqueName: \"kubernetes.io/projected/cb01fa9f-694f-423c-8d29-36832a892c8a-kube-api-access-hnz8l\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:46:32.857561 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:32.857484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:33.217437 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.217405    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb01fa9f-694f-423c-8d29-36832a892c8a" containerID="36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85" exitCode=0
Sep 07 16:46:33.217632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.217448    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"
Sep 07 16:46:33.217632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.217496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj" event={"ID":"cb01fa9f-694f-423c-8d29-36832a892c8a","Type":"ContainerDied","Data":"36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85"}
Sep 07 16:46:33.217632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.217543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj" event={"ID":"cb01fa9f-694f-423c-8d29-36832a892c8a","Type":"ContainerDied","Data":"3aa9b3b58214791077c052042e7f6aeaa9565c2cea2c77576741ecb0d1ebcc09"}
Sep 07 16:46:33.217632 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.217570    2577 scope.go:117] "RemoveContainer" containerID="36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85"
Sep 07 16:46:33.226558 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.226543    2577 scope.go:117] "RemoveContainer" containerID="36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85"
Sep 07 16:46:33.226924 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:46:33.226901    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85\": container with ID starting with 36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85 not found: ID does not exist" containerID="36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85"
Sep 07 16:46:33.226987 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.226939    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85"} err="failed to get container status \"36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85\": rpc error: code = NotFound desc = could not find container \"36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85\": container with ID starting with 36a524500b646ad9b85482b534aa75324695e7cee40bebdb958f2f26e9766a85 not found: ID does not exist"
Sep 07 16:46:33.239432 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.239410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"]
Sep 07 16:46:33.246934 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.246914    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7aca0f-867b8f448f-hb4xj"]
Sep 07 16:46:33.567628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.567554    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb01fa9f-694f-423c-8d29-36832a892c8a" path="/var/lib/kubelet/pods/cb01fa9f-694f-423c-8d29-36832a892c8a/volumes"
Sep 07 16:46:33.568032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.567901    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc661a47-b55b-405f-8900-68f55350303f" path="/var/lib/kubelet/pods/fc661a47-b55b-405f-8900-68f55350303f/volumes"
Sep 07 16:46:33.816717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:33.816682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:38.626407 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:38.626374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:38.692662 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:38.692629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:46:38.893589 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:46:38.893560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:47:55.901781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:47:55.901727    2577 scope.go:117] "RemoveContainer" containerID="5f4b93a6916d170795b28e2dacc4a89314d84050bc5b8bd1f34fe0b123d90495"
Sep 07 16:48:03.099288 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:48:03.099257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:48:04.529640 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:48:04.529605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:48:06.128863 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:48:06.128826    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:49:53.581429 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:49:53.581391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:50:01.470354 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:50:01.470314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:50:01.558373 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:50:01.558340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:50:28.349575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:50:28.349542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:50:28.504699 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:50:28.504665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:50:55.938166 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:50:55.938131    2577 scope.go:117] "RemoveContainer" containerID="d92639097ec978b86f7e053caabf5bbbfc30ac2345f4b061b3e52cb43ba31dd7"
Sep 07 16:51:05.572351 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572313    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"]
Sep 07 16:51:05.572742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572719    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy"
Sep 07 16:51:05.572742 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572733    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy"
Sep 07 16:51:05.572836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572766    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc661a47-b55b-405f-8900-68f55350303f" containerName="revoke-keys"
Sep 07 16:51:05.572836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572775    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc661a47-b55b-405f-8900-68f55350303f" containerName="revoke-keys"
Sep 07 16:51:05.572836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572793    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb01fa9f-694f-423c-8d29-36832a892c8a" containerName="payload-processing"
Sep 07 16:51:05.572836 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572800    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb01fa9f-694f-423c-8d29-36832a892c8a" containerName="payload-processing"
Sep 07 16:51:05.572960 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572858    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb01fa9f-694f-423c-8d29-36832a892c8a" containerName="payload-processing"
Sep 07 16:51:05.572960 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572869    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy"
Sep 07 16:51:05.572960 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572878    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2e6379-9c61-4533-affd-b9e3e28f9e4d" containerName="istio-proxy"
Sep 07 16:51:05.572960 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.572883    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc661a47-b55b-405f-8900-68f55350303f" containerName="revoke-keys"
Sep 07 16:51:05.575976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.575956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.579127 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.579106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1685c1e0-kserve-self-signed-certs\""
Sep 07 16:51:05.587318 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.587297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"]
Sep 07 16:51:05.628491 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.628443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:51:05.704636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.704600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-model-cache\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.704636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.704637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-dshm\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.704923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.704777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tls-certs\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.704923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.704860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxqg\" (UniqueName: \"kubernetes.io/projected/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-kube-api-access-5wxqg\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.704923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.704878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-home\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.704923 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.704893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tmp-dir\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.805819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.805783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxqg\" (UniqueName: \"kubernetes.io/projected/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-kube-api-access-5wxqg\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.805819 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.805821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-home\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.805837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tmp-dir\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.805866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-model-cache\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.805885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-dshm\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806072 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.805944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tls-certs\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806292 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.806260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-home\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.806290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-model-cache\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.806463 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.806303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tmp-dir\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.808422 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.808390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-dshm\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.808840 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.808819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tls-certs\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.815976 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.815951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxqg\" (UniqueName: \"kubernetes.io/projected/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-kube-api-access-5wxqg\") pod \"e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") " pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:05.885239 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:05.885212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:06.164563 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:06.164448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"]
Sep 07 16:51:06.171283 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:51:06.171250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf2d6c3d_1d5f_41a6_87eb_effdcc1d9557.slice/crio-7f0cbbd32f6aaead9c1bcb3f49c29eaf89c7408320e5531c65d67c65de242de1 WatchSource:0}: Error finding container 7f0cbbd32f6aaead9c1bcb3f49c29eaf89c7408320e5531c65d67c65de242de1: Status 404 returned error can't find the container with id 7f0cbbd32f6aaead9c1bcb3f49c29eaf89c7408320e5531c65d67c65de242de1
Sep 07 16:51:06.561138 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:06.561034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn" event={"ID":"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557","Type":"ContainerStarted","Data":"d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533"}
Sep 07 16:51:06.561138 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:06.561083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn" event={"ID":"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557","Type":"ContainerStarted","Data":"7f0cbbd32f6aaead9c1bcb3f49c29eaf89c7408320e5531c65d67c65de242de1"}
Sep 07 16:51:06.580771 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:06.580690    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn" podStartSLOduration=1.5806717909999999 podStartE2EDuration="1.580671791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:51:06.578915908 +0000 UTC m=+1871.460452885" watchObservedRunningTime="2026-09-07 16:51:06.580671791 +0000 UTC m=+1871.462208758"
Sep 07 16:51:15.886242 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.886200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:15.886674 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.886269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:15.898775 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.898730    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:15.935027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.934996    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"]
Sep 07 16:51:15.935384 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.935372    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy"
Sep 07 16:51:15.935428 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.935385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy"
Sep 07 16:51:15.935479 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.935470    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bffe79-62d9-4cbf-864d-fbe1067631eb" containerName="istio-proxy"
Sep 07 16:51:15.938636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.938620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:15.941097 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.941079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1685c1e0-kserve-self-signed-certs\""
Sep 07 16:51:15.947544 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:15.947520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"]
Sep 07 16:51:16.004517 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.004475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-home\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.004716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.004527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tmp-dir\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.004716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.004605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-model-cache\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.004716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.004636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-dshm\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.004716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.004668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8z45\" (UniqueName: \"kubernetes.io/projected/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-kube-api-access-j8z45\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.004716 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.004712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tls-certs\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105345 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-home\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tmp-dir\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-model-cache\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-dshm\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105545 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8z45\" (UniqueName: \"kubernetes.io/projected/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-kube-api-access-j8z45\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tls-certs\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-home\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.105790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.105740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tmp-dir\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.106717 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.106678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-model-cache\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.108498 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.108474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-dshm\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.108801 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.108779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tls-certs\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.114588 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.114566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8z45\" (UniqueName: \"kubernetes.io/projected/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-kube-api-access-j8z45\") pod \"e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") " pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.138513 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.138412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:51:16.249061 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.249023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:16.525446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.525417    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"]
Sep 07 16:51:16.527092 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:51:16.527069    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37f90f1_c40c_4dd8_8ff0_2c5c0181239f.slice/crio-c8579c4459ed7f76a1d5a1bfa4d467e848f50b1941d6fe05e9fecc45a49c85e2 WatchSource:0}: Error finding container c8579c4459ed7f76a1d5a1bfa4d467e848f50b1941d6fe05e9fecc45a49c85e2: Status 404 returned error can't find the container with id c8579c4459ed7f76a1d5a1bfa4d467e848f50b1941d6fe05e9fecc45a49c85e2
Sep 07 16:51:16.612997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.612902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k" event={"ID":"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f","Type":"ContainerStarted","Data":"983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6"}
Sep 07 16:51:16.612997 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.612950    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k" event={"ID":"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f","Type":"ContainerStarted","Data":"c8579c4459ed7f76a1d5a1bfa4d467e848f50b1941d6fe05e9fecc45a49c85e2"}
Sep 07 16:51:16.625315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.625287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:16.664101 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:16.664009    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k" podStartSLOduration=1.6639943750000001 podStartE2EDuration="1.663994375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:51:16.639776484 +0000 UTC m=+1881.521313450" watchObservedRunningTime="2026-09-07 16:51:16.663994375 +0000 UTC m=+1881.545531342"
Sep 07 16:51:26.250086 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:26.250034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:26.250086 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:26.250102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:26.262433 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:26.262398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:26.672100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:26.672073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:51:43.323032 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:43.323000    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"]
Sep 07 16:51:43.344667 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:43.344633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:51:44.771299 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:44.771267    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"]
Sep 07 16:51:44.793726 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:44.793699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:51:55.958518 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:55.958483    2577 scope.go:117] "RemoveContainer" containerID="8c90d931a4c892b3b8733002dae74e3735940ad406c436b9ef5ffdc526ee74a6"
Sep 07 16:51:55.969773 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:55.969728    2577 scope.go:117] "RemoveContainer" containerID="b96f70d3f601c57483a344ef0a89083379e3cb16aa1466bc2450897e5ee6ddce"
Sep 07 16:51:58.343584 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.343525    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn" podUID="df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" containerName="main" containerID="cri-o://d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533" gracePeriod=45
Sep 07 16:51:58.583344 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.583322    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:58.593731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.593683    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tls-certs\") pod \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") "
Sep 07 16:51:58.593731 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.593718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-home\") pod \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") "
Sep 07 16:51:58.593856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.593803    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tmp-dir\") pod \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") "
Sep 07 16:51:58.593856 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.593844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wxqg\" (UniqueName: \"kubernetes.io/projected/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-kube-api-access-5wxqg\") pod \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") "
Sep 07 16:51:58.593964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.593894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-model-cache\") pod \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") "
Sep 07 16:51:58.593964 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.593956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-dshm\") pod \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\" (UID: \"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557\") "
Sep 07 16:51:58.594073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.594048    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-home" (OuterVolumeSpecName: "home") pod "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" (UID: "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:51:58.594123 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.594084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" (UID: "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:51:58.594196 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.594177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-model-cache" (OuterVolumeSpecName: "model-cache") pod "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" (UID: "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:51:58.594314 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.594298    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:51:58.594360 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.594320    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:51:58.594360 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.594336    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:51:58.596027 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.596009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" (UID: "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:51:58.596099 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.596047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-dshm" (OuterVolumeSpecName: "dshm") pod "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" (UID: "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:51:58.596134 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.596117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-kube-api-access-5wxqg" (OuterVolumeSpecName: "kube-api-access-5wxqg") pod "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" (UID: "df2d6c3d-1d5f-41a6-87eb-effdcc1d9557"). InnerVolumeSpecName "kube-api-access-5wxqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:51:58.694922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.694890    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxqg\" (UniqueName: \"kubernetes.io/projected/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-kube-api-access-5wxqg\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:51:58.694922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.694919    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:51:58.694922 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.694933    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:51:58.818668 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.818624    2577 generic.go:356] "Generic (PLEG): container finished" podID="df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" containerID="d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533" exitCode=0
Sep 07 16:51:58.818668 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.818667    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"
Sep 07 16:51:58.818668 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.818670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn" event={"ID":"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557","Type":"ContainerDied","Data":"d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533"}
Sep 07 16:51:58.818972 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.818711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn" event={"ID":"df2d6c3d-1d5f-41a6-87eb-effdcc1d9557","Type":"ContainerDied","Data":"7f0cbbd32f6aaead9c1bcb3f49c29eaf89c7408320e5531c65d67c65de242de1"}
Sep 07 16:51:58.818972 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.818730    2577 scope.go:117] "RemoveContainer" containerID="d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533"
Sep 07 16:51:58.827572 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.827555    2577 scope.go:117] "RemoveContainer" containerID="d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533"
Sep 07 16:51:58.827875 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:51:58.827856    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533\": container with ID starting with d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533 not found: ID does not exist" containerID="d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533"
Sep 07 16:51:58.827938 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.827885    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533"} err="failed to get container status \"d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533\": rpc error: code = NotFound desc = could not find container \"d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533\": container with ID starting with d56dd4a65a0cfeaa4ca5fd2bfc5ac8c70059cc74a3a94709f7d64a4504381533 not found: ID does not exist"
Sep 07 16:51:58.840267 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.840240    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"]
Sep 07 16:51:58.846327 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:58.846266    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1685c1e0-kserve-57559794b7-bfvjn"]
Sep 07 16:51:59.567042 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:59.567011    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" path="/var/lib/kubelet/pods/df2d6c3d-1d5f-41a6-87eb-effdcc1d9557/volumes"
Sep 07 16:51:59.792033 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:51:59.791986    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k" podUID="b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" containerName="main" containerID="cri-o://983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6" gracePeriod=45
Sep 07 16:52:00.044655 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.044623    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:52:00.107996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.107963    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-model-cache\") pod \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") "
Sep 07 16:52:00.108182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108004    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-home\") pod \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") "
Sep 07 16:52:00.108182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8z45\" (UniqueName: \"kubernetes.io/projected/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-kube-api-access-j8z45\") pod \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") "
Sep 07 16:52:00.108182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tmp-dir\") pod \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") "
Sep 07 16:52:00.108182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tls-certs\") pod \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") "
Sep 07 16:52:00.108182 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-dshm\") pod \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\" (UID: \"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f\") "
Sep 07 16:52:00.108439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-home" (OuterVolumeSpecName: "home") pod "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" (UID: "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:52:00.108439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108321    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" (UID: "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:52:00.108439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108331    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" (UID: "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:52:00.108439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108428    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:00.108439 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108444    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:00.108649 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.108453    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:00.110303 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.110283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-dshm" (OuterVolumeSpecName: "dshm") pod "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" (UID: "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 16:52:00.110364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.110308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" (UID: "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:52:00.110364 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.110354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-kube-api-access-j8z45" (OuterVolumeSpecName: "kube-api-access-j8z45") pod "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" (UID: "b37f90f1-c40c-4dd8-8ff0-2c5c0181239f"). InnerVolumeSpecName "kube-api-access-j8z45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:52:00.209679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.209589    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8z45\" (UniqueName: \"kubernetes.io/projected/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-kube-api-access-j8z45\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:00.209679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.209620    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:00.209679 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.209633    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:00.831315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.831283    2577 generic.go:356] "Generic (PLEG): container finished" podID="b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" containerID="983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6" exitCode=0
Sep 07 16:52:00.831781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.831323    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"
Sep 07 16:52:00.831781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.831364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k" event={"ID":"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f","Type":"ContainerDied","Data":"983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6"}
Sep 07 16:52:00.831781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.831392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k" event={"ID":"b37f90f1-c40c-4dd8-8ff0-2c5c0181239f","Type":"ContainerDied","Data":"c8579c4459ed7f76a1d5a1bfa4d467e848f50b1941d6fe05e9fecc45a49c85e2"}
Sep 07 16:52:00.831781 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.831410    2577 scope.go:117] "RemoveContainer" containerID="983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6"
Sep 07 16:52:00.847864 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.847844    2577 scope.go:117] "RemoveContainer" containerID="983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6"
Sep 07 16:52:00.848175 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:52:00.848156    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6\": container with ID starting with 983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6 not found: ID does not exist" containerID="983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6"
Sep 07 16:52:00.848220 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.848183    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6"} err="failed to get container status \"983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6\": rpc error: code = NotFound desc = could not find container \"983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6\": container with ID starting with 983147fa8c0afdd161f38d22d992370b427cce00d8fa12437fe112aa099cfca6 not found: ID does not exist"
Sep 07 16:52:00.863698 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.863669    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"]
Sep 07 16:52:00.867193 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:00.867170    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1685c1e0-kserve-847b446895-89v6k"]
Sep 07 16:52:01.567994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:01.567962    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" path="/var/lib/kubelet/pods/b37f90f1-c40c-4dd8-8ff0-2c5c0181239f/volumes"
Sep 07 16:52:03.728627 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.728594    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"]
Sep 07 16:52:03.729013 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.729001    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" containerName="main"
Sep 07 16:52:03.729073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.729015    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" containerName="main"
Sep 07 16:52:03.729073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.729031    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" containerName="main"
Sep 07 16:52:03.729073 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.729036    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" containerName="main"
Sep 07 16:52:03.729169 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.729100    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2d6c3d-1d5f-41a6-87eb-effdcc1d9557" containerName="main"
Sep 07 16:52:03.729169 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.729111    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37f90f1-c40c-4dd8-8ff0-2c5c0181239f" containerName="main"
Sep 07 16:52:03.733547 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.733526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.737405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.737382    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj7bk\""
Sep 07 16:52:03.737405 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.737393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 16:52:03.737609 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.737449    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 16:52:03.746659 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.746615    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"]
Sep 07 16:52:03.842713 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.842674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bwl\" (UniqueName: \"kubernetes.io/projected/a0c36284-24ce-4003-b68e-ece621c591e2-kube-api-access-79bwl\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.842959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.842732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c36284-24ce-4003-b68e-ece621c591e2-openshift-service-ca\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.842959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.842841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-maas-api-tls\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.842959 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.842875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-metrics-certs\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.943814 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.943739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bwl\" (UniqueName: \"kubernetes.io/projected/a0c36284-24ce-4003-b68e-ece621c591e2-kube-api-access-79bwl\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.944001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.943875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c36284-24ce-4003-b68e-ece621c591e2-openshift-service-ca\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.944001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.943914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-maas-api-tls\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.944001 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.943940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-metrics-certs\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.944579 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.944553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c36284-24ce-4003-b68e-ece621c591e2-openshift-service-ca\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.946532 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.946513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-metrics-certs\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.946596 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.946545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-maas-api-tls\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:03.951567 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:03.951538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bwl\" (UniqueName: \"kubernetes.io/projected/a0c36284-24ce-4003-b68e-ece621c591e2-kube-api-access-79bwl\") pod \"maas-api-5868b9dbc4-z8rx4\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") " pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:04.043487 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:04.043403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:04.316102 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:04.316067    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"]
Sep 07 16:52:04.318781 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:52:04.318736    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c36284_24ce_4003_b68e_ece621c591e2.slice/crio-a463ff7c0a7bfd51be29aeaad0931bcf047668e1a8c449fc8be520b7c3376d6b WatchSource:0}: Error finding container a463ff7c0a7bfd51be29aeaad0931bcf047668e1a8c449fc8be520b7c3376d6b: Status 404 returned error can't find the container with id a463ff7c0a7bfd51be29aeaad0931bcf047668e1a8c449fc8be520b7c3376d6b
Sep 07 16:52:04.320291 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:04.320265    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 16:52:04.851955 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:04.851917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4" event={"ID":"a0c36284-24ce-4003-b68e-ece621c591e2","Type":"ContainerStarted","Data":"a463ff7c0a7bfd51be29aeaad0931bcf047668e1a8c449fc8be520b7c3376d6b"}
Sep 07 16:52:06.864051 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:06.864012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4" event={"ID":"a0c36284-24ce-4003-b68e-ece621c591e2","Type":"ContainerStarted","Data":"17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1"}
Sep 07 16:52:06.864501 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:06.864226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:06.882465 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:06.882411    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4" podStartSLOduration=1.528145199 podStartE2EDuration="3.882392309s" totalImagesPullingTime="2.35424711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:52:06.879314687 +0000 UTC m=+1931.760851657" watchObservedRunningTime="2026-09-07 16:52:06.882392309 +0000 UTC m=+1931.763929280"
Sep 07 16:52:09.636409 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:09.636361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:52:11.928230 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:11.928187    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"]
Sep 07 16:52:11.928690 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:11.928443    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4" podUID="a0c36284-24ce-4003-b68e-ece621c591e2" containerName="maas-api" containerID="cri-o://17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1" gracePeriod=30
Sep 07 16:52:11.934087 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:11.934066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:11.963560 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:11.963538    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"]
Sep 07 16:52:11.966951 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:11.966937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:11.976079 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:11.976056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"]
Sep 07 16:52:12.023199 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.023166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e12b3a8-c911-4557-83a6-3c04066c4b11-metrics-certs\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.023199 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.023202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdx8t\" (UniqueName: \"kubernetes.io/projected/3e12b3a8-c911-4557-83a6-3c04066c4b11-kube-api-access-cdx8t\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.023404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.023239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e12b3a8-c911-4557-83a6-3c04066c4b11-maas-api-tls\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.023404 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.023320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e12b3a8-c911-4557-83a6-3c04066c4b11-openshift-service-ca\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.124071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.124040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e12b3a8-c911-4557-83a6-3c04066c4b11-metrics-certs\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.124071 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.124074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdx8t\" (UniqueName: \"kubernetes.io/projected/3e12b3a8-c911-4557-83a6-3c04066c4b11-kube-api-access-cdx8t\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.124284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.124096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e12b3a8-c911-4557-83a6-3c04066c4b11-maas-api-tls\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.124284 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.124140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e12b3a8-c911-4557-83a6-3c04066c4b11-openshift-service-ca\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.124970 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.124929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e12b3a8-c911-4557-83a6-3c04066c4b11-openshift-service-ca\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.126593 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.126568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e12b3a8-c911-4557-83a6-3c04066c4b11-maas-api-tls\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.126774 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.126728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e12b3a8-c911-4557-83a6-3c04066c4b11-metrics-certs\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.132365 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.132346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdx8t\" (UniqueName: \"kubernetes.io/projected/3e12b3a8-c911-4557-83a6-3c04066c4b11-kube-api-access-cdx8t\") pod \"maas-api-5946dbd7c4-8pfvn\" (UID: \"3e12b3a8-c911-4557-83a6-3c04066c4b11\") " pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.176454 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.176427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:12.224855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.224778    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c36284-24ce-4003-b68e-ece621c591e2-openshift-service-ca\") pod \"a0c36284-24ce-4003-b68e-ece621c591e2\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") "
Sep 07 16:52:12.224855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.224808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-metrics-certs\") pod \"a0c36284-24ce-4003-b68e-ece621c591e2\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") "
Sep 07 16:52:12.224855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.224830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-maas-api-tls\") pod \"a0c36284-24ce-4003-b68e-ece621c591e2\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") "
Sep 07 16:52:12.224855 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.224855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79bwl\" (UniqueName: \"kubernetes.io/projected/a0c36284-24ce-4003-b68e-ece621c591e2-kube-api-access-79bwl\") pod \"a0c36284-24ce-4003-b68e-ece621c591e2\" (UID: \"a0c36284-24ce-4003-b68e-ece621c591e2\") "
Sep 07 16:52:12.225207 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.225182    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c36284-24ce-4003-b68e-ece621c591e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0c36284-24ce-4003-b68e-ece621c591e2" (UID: "a0c36284-24ce-4003-b68e-ece621c591e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 16:52:12.227193 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.227167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0c36284-24ce-4003-b68e-ece621c591e2" (UID: "a0c36284-24ce-4003-b68e-ece621c591e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:52:12.227286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.227226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a0c36284-24ce-4003-b68e-ece621c591e2" (UID: "a0c36284-24ce-4003-b68e-ece621c591e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 16:52:12.227336 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.227275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c36284-24ce-4003-b68e-ece621c591e2-kube-api-access-79bwl" (OuterVolumeSpecName: "kube-api-access-79bwl") pod "a0c36284-24ce-4003-b68e-ece621c591e2" (UID: "a0c36284-24ce-4003-b68e-ece621c591e2"). InnerVolumeSpecName "kube-api-access-79bwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 16:52:12.302169 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.302126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:12.326527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.326498    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bwl\" (UniqueName: \"kubernetes.io/projected/a0c36284-24ce-4003-b68e-ece621c591e2-kube-api-access-79bwl\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:12.326527 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.326525    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c36284-24ce-4003-b68e-ece621c591e2-openshift-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:12.326695 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.326536    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-metrics-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:12.326695 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.326544    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c36284-24ce-4003-b68e-ece621c591e2-maas-api-tls\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Sep 07 16:52:12.573636 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.573610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"]
Sep 07 16:52:12.575410 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:52:12.575379    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e12b3a8_c911_4557_83a6_3c04066c4b11.slice/crio-cc141b665e3a05fe74107be9b02454369e768b3d93451728bd927222b9749052 WatchSource:0}: Error finding container cc141b665e3a05fe74107be9b02454369e768b3d93451728bd927222b9749052: Status 404 returned error can't find the container with id cc141b665e3a05fe74107be9b02454369e768b3d93451728bd927222b9749052
Sep 07 16:52:12.894465 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.894432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn" event={"ID":"3e12b3a8-c911-4557-83a6-3c04066c4b11","Type":"ContainerStarted","Data":"cc141b665e3a05fe74107be9b02454369e768b3d93451728bd927222b9749052"}
Sep 07 16:52:12.895661 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.895629    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0c36284-24ce-4003-b68e-ece621c591e2" containerID="17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1" exitCode=0
Sep 07 16:52:12.895790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.895668    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"
Sep 07 16:52:12.895790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.895673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4" event={"ID":"a0c36284-24ce-4003-b68e-ece621c591e2","Type":"ContainerDied","Data":"17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1"}
Sep 07 16:52:12.895790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.895722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4" event={"ID":"a0c36284-24ce-4003-b68e-ece621c591e2","Type":"ContainerDied","Data":"a463ff7c0a7bfd51be29aeaad0931bcf047668e1a8c449fc8be520b7c3376d6b"}
Sep 07 16:52:12.895790 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.895744    2577 scope.go:117] "RemoveContainer" containerID="17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1"
Sep 07 16:52:12.912711 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.912696    2577 scope.go:117] "RemoveContainer" containerID="17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1"
Sep 07 16:52:12.913066 ip-10-0-129-87 kubenswrapper[2577]: E0907 16:52:12.913044    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1\": container with ID starting with 17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1 not found: ID does not exist" containerID="17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1"
Sep 07 16:52:12.913151 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.913073    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1"} err="failed to get container status \"17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1\": rpc error: code = NotFound desc = could not find container \"17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1\": container with ID starting with 17b3b596bad0552b98f1bcae8715bcd8fd802b4cd2758c3873528273621fadc1 not found: ID does not exist"
Sep 07 16:52:12.922329 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.919280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"]
Sep 07 16:52:12.924013 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:12.923994    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5868b9dbc4-z8rx4"]
Sep 07 16:52:13.567603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:13.567570    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c36284-24ce-4003-b68e-ece621c591e2" path="/var/lib/kubelet/pods/a0c36284-24ce-4003-b68e-ece621c591e2/volumes"
Sep 07 16:52:13.902768 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:13.902726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn" event={"ID":"3e12b3a8-c911-4557-83a6-3c04066c4b11","Type":"ContainerStarted","Data":"d7deaca401b25c229e85f0cd571909b2a05bddf135e1b7d2334e432941d1d5c8"}
Sep 07 16:52:13.902944 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:13.902873    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:19.917196 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:19.917158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn"
Sep 07 16:52:19.934806 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:19.934729    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5946dbd7c4-8pfvn" podStartSLOduration=8.586716715 podStartE2EDuration="8.934713483s" totalImagesPullingTime="347.996768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:52:13.923296607 +0000 UTC m=+1938.804833575" watchObservedRunningTime="2026-09-07 16:52:19.934713483 +0000 UTC m=+1944.816250451"
Sep 07 16:52:38.525475 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:38.525439    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55cd58ff95-c565w_6e8c7ee9-8c06-4f0b-8448-9c30c672ea66/authorino/0.log"
Sep 07 16:52:44.223628 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:44.223594    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-b67gx_fab5e120-7b65-420c-a1ab-84cc4f202814/copy-manifests/0.log"
Sep 07 16:52:44.230153 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:44.230114    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-b67gx_fab5e120-7b65-420c-a1ab-84cc4f202814/manager/0.log"
Sep 07 16:52:44.700961 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:44.700933    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5946dbd7c4-8pfvn_3e12b3a8-c911-4557-83a6-3c04066c4b11/maas-api/0.log"
Sep 07 16:52:44.915264 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:44.915237    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813310-65b47_b76c73ab-06b3-442b-bee7-446795081e41/cleanup/0.log"
Sep 07 16:52:45.039216 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:45.039147    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813325-t9tnn_8830d650-9b30-420d-b523-c408a6c297e7/cleanup/0.log"
Sep 07 16:52:45.147151 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:45.147119    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-k4rps_688bddf6-e99a-43b8-8b64-65cea3eae249/postgres/0.log"
Sep 07 16:52:45.889603 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:45.889569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55cd58ff95-c565w_6e8c7ee9-8c06-4f0b-8448-9c30c672ea66/authorino/0.log"
Sep 07 16:52:46.559639 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:46.559608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-l6ssx_fc41bd89-0b91-440e-be33-151ae38f506b/manager/0.log"
Sep 07 16:52:47.220176 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:47.220140    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6797cb654d-6kdxj_0478b80e-6252-47e7-8559-4bff15e9d972/kube-auth-proxy/0.log"
Sep 07 16:52:48.122838 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:48.122805    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2rll7_99a69e34-7ed4-4d99-a0fc-5874de5aa171/main/0.log"
Sep 07 16:52:53.488947 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:53.488912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:52:53.561315 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:53.561280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log" limit=1024
Sep 07 16:52:55.993663 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:55.993625    2577 scope.go:117] "RemoveContainer" containerID="9dc83fe165c074939f171b2671e411d28a064d8cf517818a56c81175e8e40e1d"
Sep 07 16:52:56.424358 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:56.424332    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n2rmj_044272e8-5427-435e-acbe-b6b090dde0d3/global-pull-secret-syncer/0.log"
Sep 07 16:52:56.496180 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:56.496155    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d8lft_19ece502-a089-48d7-b9fd-6c4a22ba9ab0/konnectivity-agent/0.log"
Sep 07 16:52:56.599023 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:52:56.598989    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-87.ec2.internal_115763896ffc7ec5e812873cfb64a09e/haproxy/0.log"
Sep 07 16:53:01.785729 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:01.785694    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55cd58ff95-c565w_6e8c7ee9-8c06-4f0b-8448-9c30c672ea66/authorino/0.log"
Sep 07 16:53:02.038055 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:02.037945    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-l6ssx_fc41bd89-0b91-440e-be33-151ae38f506b/manager/0.log"
Sep 07 16:53:03.323068 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.323035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/alertmanager/0.log"
Sep 07 16:53:03.342552 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.342520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/config-reloader/0.log"
Sep 07 16:53:03.361642 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.361606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/kube-rbac-proxy-web/0.log"
Sep 07 16:53:03.384626 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.384601    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/kube-rbac-proxy/0.log"
Sep 07 16:53:03.410557 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.410526    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/kube-rbac-proxy-metric/0.log"
Sep 07 16:53:03.438647 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.438613    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/prom-label-proxy/0.log"
Sep 07 16:53:03.465719 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.465695    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5172ec97-144e-46c1-b07e-73f3885cb3f8/init-config-reloader/0.log"
Sep 07 16:53:03.574926 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.574898    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-x5vz2_cb611977-d6ac-4192-8c34-bdcd6fa8b612/kube-state-metrics/0.log"
Sep 07 16:53:03.598087 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.598045    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-x5vz2_cb611977-d6ac-4192-8c34-bdcd6fa8b612/kube-rbac-proxy-main/0.log"
Sep 07 16:53:03.621057 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.621028    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-x5vz2_cb611977-d6ac-4192-8c34-bdcd6fa8b612/kube-rbac-proxy-self/0.log"
Sep 07 16:53:03.646251 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.646220    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56df6586f6-sbpsp_76f8088f-2f88-4048-a107-715c505b8027/metrics-server/0.log"
Sep 07 16:53:03.669994 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.669964    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-hjq5m_1ef743c7-f610-46b1-a8cd-f5696229f36b/monitoring-plugin/0.log"
Sep 07 16:53:03.854629 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.854523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhvmn_e22673d7-fc8d-4890-8251-e32f7d7f7de1/node-exporter/0.log"
Sep 07 16:53:03.871678 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.871656    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhvmn_e22673d7-fc8d-4890-8251-e32f7d7f7de1/kube-rbac-proxy/0.log"
Sep 07 16:53:03.898322 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.898293    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhvmn_e22673d7-fc8d-4890-8251-e32f7d7f7de1/init-textfile/0.log"
Sep 07 16:53:03.919580 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.919540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-j8vcm_289f2444-775c-48d4-956b-14d260dca36f/kube-rbac-proxy-main/0.log"
Sep 07 16:53:03.943542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.943509    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-j8vcm_289f2444-775c-48d4-956b-14d260dca36f/kube-rbac-proxy-self/0.log"
Sep 07 16:53:03.962996 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.962968    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-j8vcm_289f2444-775c-48d4-956b-14d260dca36f/openshift-state-metrics/0.log"
Sep 07 16:53:03.999406 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:03.999381    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/prometheus/0.log"
Sep 07 16:53:04.017573 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.017553    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/config-reloader/0.log"
Sep 07 16:53:04.038230 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.038208    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/thanos-sidecar/0.log"
Sep 07 16:53:04.058020 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.057980    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/kube-rbac-proxy-web/0.log"
Sep 07 16:53:04.076286 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.076250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/kube-rbac-proxy/0.log"
Sep 07 16:53:04.105192 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.105116    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/kube-rbac-proxy-thanos/0.log"
Sep 07 16:53:04.124281 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.124243    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8c8709d2-fd14-4dfb-987f-b353237b61ec/init-config-reloader/0.log"
Sep 07 16:53:04.155367 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.155340    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-qh8fx_270b6c67-86a2-4e5b-8322-6a1a10072ca9/prometheus-operator/0.log"
Sep 07 16:53:04.169787 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.169766    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-qh8fx_270b6c67-86a2-4e5b-8322-6a1a10072ca9/kube-rbac-proxy/0.log"
Sep 07 16:53:04.226707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.226643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bfffc7ff4-nbzpp_8eb07a7c-6eb8-4b7c-841a-4f41304e446e/telemeter-client/0.log"
Sep 07 16:53:04.252372 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.252342    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bfffc7ff4-nbzpp_8eb07a7c-6eb8-4b7c-841a-4f41304e446e/reload/0.log"
Sep 07 16:53:04.272515 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.272488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bfffc7ff4-nbzpp_8eb07a7c-6eb8-4b7c-841a-4f41304e446e/kube-rbac-proxy/0.log"
Sep 07 16:53:04.307484 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.307451    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5564f7d89f-cwhn7_713b580e-0a97-45d9-9dbd-c72372688503/thanos-query/0.log"
Sep 07 16:53:04.333809 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.333784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5564f7d89f-cwhn7_713b580e-0a97-45d9-9dbd-c72372688503/kube-rbac-proxy-web/0.log"
Sep 07 16:53:04.351977 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.351952    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5564f7d89f-cwhn7_713b580e-0a97-45d9-9dbd-c72372688503/kube-rbac-proxy/0.log"
Sep 07 16:53:04.371276 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.371200    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5564f7d89f-cwhn7_713b580e-0a97-45d9-9dbd-c72372688503/prom-label-proxy/0.log"
Sep 07 16:53:04.388849 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.388821    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5564f7d89f-cwhn7_713b580e-0a97-45d9-9dbd-c72372688503/kube-rbac-proxy-rules/0.log"
Sep 07 16:53:04.407845 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:04.407823    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5564f7d89f-cwhn7_713b580e-0a97-45d9-9dbd-c72372688503/kube-rbac-proxy-metrics/0.log"
Sep 07 16:53:06.217784 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.217736    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"]
Sep 07 16:53:06.218178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.218162    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c36284-24ce-4003-b68e-ece621c591e2" containerName="maas-api"
Sep 07 16:53:06.218178 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.218178    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c36284-24ce-4003-b68e-ece621c591e2" containerName="maas-api"
Sep 07 16:53:06.218289 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.218254    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c36284-24ce-4003-b68e-ece621c591e2" containerName="maas-api"
Sep 07 16:53:06.221672 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.221653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.224008 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.223986    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v74r\"/\"kube-root-ca.crt\""
Sep 07 16:53:06.225181 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.225161    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v74r\"/\"openshift-service-ca.crt\""
Sep 07 16:53:06.225313 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.225161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5v74r\"/\"default-dockercfg-trgjt\""
Sep 07 16:53:06.230368 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.230342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"]
Sep 07 16:53:06.325519 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.325472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-proc\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.325519 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.325520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpspm\" (UniqueName: \"kubernetes.io/projected/bf01ae7e-7a34-4253-861d-118a8d2e05c9-kube-api-access-gpspm\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.325785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.325540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-podres\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.325785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.325687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-lib-modules\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.325785 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.325738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-sys\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426703 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-sys\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-proc\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpspm\" (UniqueName: \"kubernetes.io/projected/bf01ae7e-7a34-4253-861d-118a8d2e05c9-kube-api-access-gpspm\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-podres\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-sys\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-proc\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.426900 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-lib-modules\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.427100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.426937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-podres\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.427100 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.427003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf01ae7e-7a34-4253-861d-118a8d2e05c9-lib-modules\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.436596 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.436564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpspm\" (UniqueName: \"kubernetes.io/projected/bf01ae7e-7a34-4253-861d-118a8d2e05c9-kube-api-access-gpspm\") pod \"perf-node-gather-daemonset-pndv9\" (UID: \"bf01ae7e-7a34-4253-861d-118a8d2e05c9\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.532853 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.532744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:06.804891 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:06.804867    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"]
Sep 07 16:53:06.807450 ip-10-0-129-87 kubenswrapper[2577]: W0907 16:53:06.807426    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf01ae7e_7a34_4253_861d_118a8d2e05c9.slice/crio-6b0aff00706c1fd1ab7093e659fbaf53fec8dc5c09be52d5a6192bbd1f620f46 WatchSource:0}: Error finding container 6b0aff00706c1fd1ab7093e659fbaf53fec8dc5c09be52d5a6192bbd1f620f46: Status 404 returned error can't find the container with id 6b0aff00706c1fd1ab7093e659fbaf53fec8dc5c09be52d5a6192bbd1f620f46
Sep 07 16:53:07.168859 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:07.168827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9" event={"ID":"bf01ae7e-7a34-4253-861d-118a8d2e05c9","Type":"ContainerStarted","Data":"7b1c62143e6f6aa71943010be0ca9196677e194d5d8923d79869178906d8b643"}
Sep 07 16:53:07.169028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:07.168897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9" event={"ID":"bf01ae7e-7a34-4253-861d-118a8d2e05c9","Type":"ContainerStarted","Data":"6b0aff00706c1fd1ab7093e659fbaf53fec8dc5c09be52d5a6192bbd1f620f46"}
Sep 07 16:53:07.169028 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:07.168931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:07.183905 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:07.183860    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9" podStartSLOduration=1.183846919 podStartE2EDuration="1.183846919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:53:07.183365288 +0000 UTC m=+1992.064902259" watchObservedRunningTime="2026-09-07 16:53:07.183846919 +0000 UTC m=+1992.065383887"
Sep 07 16:53:08.464643 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:08.464610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgfc2_fa6fd336-0b62-4b47-aa45-69b561465ad3/dns/0.log"
Sep 07 16:53:08.482865 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:08.482827    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgfc2_fa6fd336-0b62-4b47-aa45-69b561465ad3/kube-rbac-proxy/0.log"
Sep 07 16:53:08.537796 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:08.537772    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hsq5z_348bb70e-a113-4727-b293-a3a369b064ab/dns-node-resolver/0.log"
Sep 07 16:53:09.119279 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:09.119246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vprsh_0f93a8e0-8f59-41cd-acd2-8e4baebfa6dc/node-ca/0.log"
Sep 07 16:53:10.606590 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:10.606563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6797cb654d-6kdxj_0478b80e-6252-47e7-8559-4bff15e9d972/kube-auth-proxy/0.log"
Sep 07 16:53:11.560163 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:11.560131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5rwnq_24c58aca-c978-4eba-a4a0-6645b48e2f38/serve-healthcheck-canary/0.log"
Sep 07 16:53:12.187264 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:12.187229    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v8n78_4e4be2ed-750a-4042-a911-f20c3bb88f0b/kube-rbac-proxy/0.log"
Sep 07 16:53:12.206410 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:12.206382    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v8n78_4e4be2ed-750a-4042-a911-f20c3bb88f0b/exporter/0.log"
Sep 07 16:53:12.224430 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:12.224398    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v8n78_4e4be2ed-750a-4042-a911-f20c3bb88f0b/extractor/0.log"
Sep 07 16:53:13.184542 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:13.184518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-pndv9"
Sep 07 16:53:14.662462 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:14.662433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-b67gx_fab5e120-7b65-420c-a1ab-84cc4f202814/manager/0.log"
Sep 07 16:53:14.678707 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:14.678682    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-b67gx_fab5e120-7b65-420c-a1ab-84cc4f202814/copy-manifests/0.log"
Sep 07 16:53:16.137924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:16.137891    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57d49c86b4-2pmxs_afc46a41-520b-4a4b-abf3-33d54eb9dc25/manager/0.log"
Sep 07 16:53:17.099381 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:17.099350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6797cb654d-6kdxj_0478b80e-6252-47e7-8559-4bff15e9d972/kube-auth-proxy/0.log"
Sep 07 16:53:19.234306 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:19.234263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6797cb654d-6kdxj_0478b80e-6252-47e7-8559-4bff15e9d972/kube-auth-proxy/0.log"
Sep 07 16:53:22.313924 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.313876    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6q2fk_c890a8e3-57f5-402f-86cb-460f4d946769/kube-multus/0.log"
Sep 07 16:53:22.595701 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.595679    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/kube-multus-additional-cni-plugins/0.log"
Sep 07 16:53:22.613575 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.613550    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/egress-router-binary-copy/0.log"
Sep 07 16:53:22.630174 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.630151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/cni-plugins/0.log"
Sep 07 16:53:22.646794 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.646720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/bond-cni-plugin/0.log"
Sep 07 16:53:22.663155 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.663128    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/routeoverride-cni/0.log"
Sep 07 16:53:22.679680 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.679658    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/whereabouts-cni-bincopy/0.log"
Sep 07 16:53:22.699330 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.699309    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4bpc_15b1d5eb-c920-4f26-8732-93ac130b7ca3/whereabouts-cni/0.log"
Sep 07 16:53:22.770887 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.770864    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9pgv4_55f00fc3-8fcf-4f2c-b951-15e809f38527/network-metrics-daemon/0.log"
Sep 07 16:53:22.789446 ip-10-0-129-87 kubenswrapper[2577]: I0907 16:53:22.789418    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9pgv4_55f00fc3-8fcf-4f2c-b951-15e809f38527/kube-rbac-proxy/0.log"