Sep 18 18:00:28.357303 ip-10-0-136-64 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 18:00:31.734597 ip-10-0-136-64 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:00:31.734597 ip-10-0-136-64 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 18:00:31.734597 ip-10-0-136-64 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:00:31.734597 ip-10-0-136-64 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 18:00:31.734597 ip-10-0-136-64 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:00:31.735324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.734670    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 18:00:31.737652 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737635    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:00:31.737652 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737651    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737656    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737660    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737663    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737666    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737669    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737672    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737681    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737684    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737686    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737689    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737691    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737694    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737696    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737699    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737701    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737704    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737707    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737711    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737714    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:00:31.737713 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737718    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737721    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737725    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737728    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737731    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737734    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737736    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737739    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737742    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737744    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737747    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737749    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737752    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737754    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737757    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737759    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737762    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737764    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737767    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737770    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:00:31.738240 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737772    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737775    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737777    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737780    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737782    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737784    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737787    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737789    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737792    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737794    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737796    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737799    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737803    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737806    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737810    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737813    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737815    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737818    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737820    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:00:31.739042 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737823    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737826    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737828    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737831    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737833    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737836    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737838    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737840    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737843    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737845    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737847    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737850    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737853    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737856    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737858    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737861    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737863    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737866    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737868    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737871    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:00:31.739827 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737873    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737876    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737878    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737880    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737883    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737885    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737890    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737894    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737897    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737900    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737902    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737905    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737907    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737910    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737912    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737915    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737918    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737920    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.737923    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740045    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:00:31.740725 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740063    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740070    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740075    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740079    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740084    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740092    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740100    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740105    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740111    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740117    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740122    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740126    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740130    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740135    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740139    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740143    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740147    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740151    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740155    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:00:31.741614 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740164    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740168    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740173    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740177    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740182    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740186    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740190    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740194    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740198    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740202    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740206    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740211    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740217    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740221    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740225    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740230    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740234    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740240    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740244    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:00:31.742548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740248    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740253    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740257    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740261    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740266    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740270    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740274    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740277    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740282    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740286    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740290    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740294    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740299    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740303    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740307    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740311    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740316    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740322    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740327    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740331    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:00:31.743427 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740337    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740342    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740346    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740350    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740355    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740359    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740363    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740367    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740372    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740376    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740380    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740384    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740388    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740392    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740396    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740400    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740405    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740409    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740413    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740417    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:00:31.743965 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740421    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740425    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740429    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740433    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740438    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740442    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740446    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740450    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740454    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740458    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740462    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740466    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740473    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740478    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740483    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740488    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740493    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740498    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.740504    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741151    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:00:31.744502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741161    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741167    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741171    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741176    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741181    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741185    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741190    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741194    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741198    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741203    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741207    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741211    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741216    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741220    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741224    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741230    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741234    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741238    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741243    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741247    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:00:31.745249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741251    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741256    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741260    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741264    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741268    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741272    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741276    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741280    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741285    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741290    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741294    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741298    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741303    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741307    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741312    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741316    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741320    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741325    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741349    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:00:31.745794 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741355    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741362    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741369    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741373    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741378    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741383    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741388    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741392    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741396    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741401    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741406    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741411    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741415    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741419    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741424    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741428    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741433    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741438    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741442    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:00:31.746327 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741446    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741450    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741459    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741466    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741472    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741477    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741481    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741485    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741489    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741493    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741497    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741501    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741505    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741510    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741514    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741520    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741524    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741528    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741532    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:00:31.746845 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741536    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741541    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741545    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741549    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741554    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741558    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741563    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741567    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741571    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741575    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741579    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741583    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741587    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741591    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741595    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741603    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741610    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741615    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741619    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:00:31.747340 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741623    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.741627    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741727    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741782    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741790    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741798    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741805    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741810    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741818    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741824    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741830    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741835    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741840    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741845    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741851    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741855    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741860    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741865    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741872    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741877    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741884    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741889    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741894    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741899    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 18:00:31.747809 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741904    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741910    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741915    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741920    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741927    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741944    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741949    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741954    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741959    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741964    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741970    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741975    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741980    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741984    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741989    2580 flags.go:64] FLAG: --enable-server="true"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.741994    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742000    2580 flags.go:64] FLAG: --event-burst="100"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742005    2580 flags.go:64] FLAG: --event-qps="50"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742009    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742014    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742019    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742044    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742049    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742053    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742058    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 18 18:00:31.748440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742063    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742068    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742073    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742078    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742082    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742086    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742090    2580 flags.go:64] FLAG: --feature-gates=""
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742096    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742101    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742106    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742111    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742118    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742125    2580 flags.go:64] FLAG: --help="false"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742129    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-136-64.ec2.internal"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742134    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742139    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742143    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742149    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742154    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742160    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742166    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742172    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742176    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 18:00:31.749042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742181    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742186    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742191    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742196    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742201    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742206    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742210    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742215    2580 flags.go:64] FLAG: --lock-file=""
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742219    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742224    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742228    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742236    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742242    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742247    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742251    2580 flags.go:64] FLAG: --logging-format="text"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742256    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742261    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742266    2580 flags.go:64] FLAG: --manifest-url=""
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742270    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742277    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742284    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742292    2580 flags.go:64] FLAG: --max-pods="110"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742297    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742301    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742306    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 18:00:31.749729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742311    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742316    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742320    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742326    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742337    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742343    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742348    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742353    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742358    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742366    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742371    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742376    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742380    2580 flags.go:64] FLAG: --port="10250"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742385    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742390    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-095f322ff9d07067e"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742395    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742399    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742404    2580 flags.go:64] FLAG: --register-node="true"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742408    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742414    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742419    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742424    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742429    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742434    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742439    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 18:00:31.750454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742444    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742448    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742453    2580 flags.go:64] FLAG: --runonce="false"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742462    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742467    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742472    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742477    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742481    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742485    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742490    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742495    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742499    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742503    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742507    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742511    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742516    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742520    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742525    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742533    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742537    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742542    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742550    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742555    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742559    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742563    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 18:00:31.751157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742567    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742571    2580 flags.go:64] FLAG: --v="2"
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742585    2580 flags.go:64] FLAG: --version="false"
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742593    2580 flags.go:64] FLAG: --vmodule=""
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742598    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.742604    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742783    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742790    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742795    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742799    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742804    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742808    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742812    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742816    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742823    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742829    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742833    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742838    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742844    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742850    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:00:31.751779 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742855    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742860    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742865    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742869    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742873    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742877    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742881    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742885    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742889    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742893    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742897    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742901    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742905    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742909    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742912    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742916    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742921    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742926    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742930    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742933    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:00:31.752289 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742937    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742941    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742946    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742952    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742958    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742963    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742968    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742972    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742977    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742981    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742986    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742990    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742994    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.742997    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743001    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743005    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743009    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743013    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743018    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743043    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:00:31.752770 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743048    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743052    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743056    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743060    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743065    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743069    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743072    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743076    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743080    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743087    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743091    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743095    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743099    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743103    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743107    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743112    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743116    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743121    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743125    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:00:31.753278 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743129    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743133    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743137    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743141    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743145    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743150    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743153    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743158    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743162    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743166    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743170    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743174    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743178    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743182    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743187    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743191    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743194    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743198    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743202    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743207    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:00:31.753738 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743211    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743215    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743219    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743224    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743228    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.743232    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.743241    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.751106    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.751122    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.751143    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.751152    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751207    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751213    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751217    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751220    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:00:31.754258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751223    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751225    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751228    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751231    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751234    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751236    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751239    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751241    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751244    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751247    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751250    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751252    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751255    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751258    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751261    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751265    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751268    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751272    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751275    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751278    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:00:31.754635 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751281    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751284    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751301    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751305    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751308    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751312    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751315    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751318    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751320    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751323    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751326    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751329    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751331    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751334    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751337    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751339    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751342    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751345    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751347    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751350    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:00:31.755144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751353    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751355    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751358    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751361    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751364    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751367    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751370    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751374    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751377    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751380    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751383    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751385    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751388    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751390    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751393    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751395    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751398    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751400    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751403    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:00:31.755663 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751406    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751408    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751412    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751415    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751418    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751421    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751423    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751426    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751428    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751431    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751452    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751455    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751457    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751460    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751462    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751465    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751467    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751471    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751473    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751475    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:00:31.756162 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751478    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751481    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751484    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751486    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751488    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751491    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751494    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751496    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751498    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751501    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751503    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751506    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751508    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751511    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751513    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:00:31.756648 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751516    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.751521    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751643    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751647    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751650    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751653    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751656    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751658    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751661    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751663    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751666    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751669    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751671    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751673    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751676    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:00:31.757018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751678    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751681    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751683    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751687    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751691    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751693    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751696    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751699    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751702    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751705    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751707    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751710    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751712    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751715    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751717    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751720    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751723    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751725    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751727    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:00:31.757403 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751730    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751732    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751735    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751737    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751740    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751742    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751745    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751747    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751750    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751752    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751755    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751757    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751760    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751762    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751765    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751767    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751770    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751773    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751775    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751777    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:00:31.757867 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751780    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751783    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751785    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751788    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751791    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751794    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751796    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751799    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751801    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751804    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751807    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751809    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751812    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751815    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751818    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751821    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751823    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751826    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751828    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751831    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:00:31.758379 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751833    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751836    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751839    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751841    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751843    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751846    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751848    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751851    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751854    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751856    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751858    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751861    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751863    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751866    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751868    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751870    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751873    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751875    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751878    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751880    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:00:31.758869 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751882    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751885    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751888    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751890    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751894    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751897    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:31.751900    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.751905    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.752011    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.754085    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.754954    2580 server.go:1019] "Starting client certificate rotation"
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.755054    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.755083    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 18:00:31.759976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.757211    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 18:00:31.760326 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.758916    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 18:00:31.770260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.770236    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 18 18:00:31.772555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.772535    2580 log.go:25] "Validated CRI v1 image API"
Sep 18 18:00:31.773429 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.773408    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 18:00:31.775576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.775560    2580 fs.go:135] Filesystem UUIDs: map[4e33d7f2-5d09-44de-a1ec-55b73ee5d3f5:/dev/nvme0n1p4 5711f596-0442-45fe-bf15-bec8930f46a9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 18 18:00:31.775643 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.775576    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 18:00:31.780060 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.779938    2580 manager.go:216] Machine: {Timestamp:2026-09-18 18:00:31.778705761 +0000 UTC m=+1.930580588 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098926 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b276356f04a8f7c63088859f38efa SystemUUID:ec2b2763-56f0-4a8f-7c63-088859f38efa BootID:0af5dedf-ad42-4af4-9a38-896d7f3b686f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:3c:0d:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:3c:0d:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:79:d1:fc:07:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 18:00:31.780060 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.780053    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 18:00:31.780167 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.780156    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 18:00:31.781600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.781579    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 18:00:31.781720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.781602    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 18:00:31.782825 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.782815    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 18:00:31.782861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.782828    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 18:00:31.782861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.782835    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 18:00:31.782861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.782845    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 18:00:31.784210 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.784201    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 18:00:31.788190 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788179    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 18:00:31.788298 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788289    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 18:00:31.788327 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788321    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 18:00:31.788357 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788331    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 18:00:31.788357 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788347    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 18:00:31.788357 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788356    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 18 18:00:31.788437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788366    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 18:00:31.788687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788676    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 18:00:31.788768 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.788724    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 18:00:31.790489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.790470    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 18:00:31.792717 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.792704    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 18:00:31.792789 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.792734    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 18:00:31.793914 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793903    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 18:00:31.793967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793920    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 18:00:31.793967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793926    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 18:00:31.793967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793931    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 18:00:31.793967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793953    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 18:00:31.793967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793962    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793979    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793985    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793992    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.793997    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.794006    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.794011    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.794054    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 18:00:31.794127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.794060    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 18:00:31.797603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.797589    2580 server.go:1268] "Started kubelet"
Sep 18 18:00:31.797920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.797897    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 18:00:31.798201 ip-10-0-136-64 systemd[1]: Started Kubernetes Kubelet.
Sep 18 18:00:31.798749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.798241    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 18:00:31.798749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.798312    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 18:00:31.799057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.799009    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 18:00:31.799482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.799467    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 18:00:31.827876 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.827843    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 18:00:31.827971 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.827910    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 18:00:31.828080 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.828053    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 18:00:31.828675 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.828652    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 18:00:31.832016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.831984    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 18:00:31.832933 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.832126    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18d67c2c5679a9c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-09-18 18:00:31.797569987 +0000 UTC m=+1.949444814,LastTimestamp:2026-09-18 18:00:31.797569987 +0000 UTC m=+1.949444814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Sep 18 18:00:31.834038 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.834006    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 18:00:31.834204 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.834145    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 18:00:31.834357 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.834313    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:31.834876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.834819    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 18:00:31.834993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.834975    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 18:00:31.835068 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835010    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 18:00:31.835122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835110    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 18:00:31.835122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835118    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 18:00:31.835198 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835151    2580 factory.go:55] Registering systemd factory
Sep 18 18:00:31.835198 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835162    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 18 18:00:31.835373 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835360    2580 factory.go:153] Registering CRI-O factory
Sep 18 18:00:31.835419 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835378    2580 factory.go:223] Registration of the crio container factory successfully
Sep 18 18:00:31.835451 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835425    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 18:00:31.835451 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835443    2580 factory.go:103] Registering Raw factory
Sep 18 18:00:31.835504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835459    2580 manager.go:1196] Started watching for new ooms in manager
Sep 18 18:00:31.835881 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.835865    2580 manager.go:349] Starting recovery of all containers
Sep 18 18:00:31.840587 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.840564    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 18:00:31.840703 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.840602    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 18:00:31.844400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.844385    2580 manager.go:354] Recovery completed
Sep 18 18:00:31.848958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.848926    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-drsnt"
Sep 18 18:00:31.849601 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.849587    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:31.851218 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851202    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:31.851298 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851229    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:31.851298 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851244    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:31.851722 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851711    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 18:00:31.851722 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851722    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 18:00:31.851823 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851739    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 18:00:31.851823 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.851772    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:31.852454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.852437    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:31.852515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.852460    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:31.852515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.852472    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:31.853051 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.852969    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18d67c2c59ac45d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-09-18 18:00:31.85121839 +0000 UTC m=+2.003093219,LastTimestamp:2026-09-18 18:00:31.85121839 +0000 UTC m=+2.003093219,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Sep 18 18:00:31.853914 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.853902    2580 policy_none.go:49] "None policy: Start"
Sep 18 18:00:31.853983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.853918    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 18:00:31.853983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.853928    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 18:00:31.855989 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.855978    2580 policy_none.go:47] "Start"
Sep 18 18:00:31.862668 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.862529    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18d67c2c59ac8944  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-09-18 18:00:31.851235652 +0000 UTC m=+2.003110482,LastTimestamp:2026-09-18 18:00:31.851235652 +0000 UTC m=+2.003110482,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Sep 18 18:00:31.865083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.865068    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-drsnt"
Sep 18 18:00:31.900576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.900558    2580 manager.go:337] "Starting Device Plugin manager"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.900596    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.900610    2580 server.go:87] "Starting device plugin registration server"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.900866    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.900888    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.900963    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.901084    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.901094    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.901588    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.901640    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.902112    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.903479    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.903521    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.903539    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:31.903573    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 18:00:31.907540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:31.905757    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:00:32.001617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.001561    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:32.002355 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.002340    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:32.002426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.002373    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:32.002426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.002384    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:32.002426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.002405    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.004508 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.004491    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"]
Sep 18 18:00:32.004552 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.004547    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:32.005237 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.005216    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:32.005328 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.005246    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:32.005328 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.005258    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:32.007500 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.007488    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:32.007611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.007597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.007650 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.007628    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:32.008233 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008209    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:32.008233 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008236    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:32.008386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008249    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:32.008386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008281    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:32.008386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008311    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:32.008386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008324    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:32.008616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.008599    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.008692 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.008618    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-64.ec2.internal\": node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.010572 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.010556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.010621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.010588    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:00:32.011232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.011217    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:00:32.011286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.011246    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:00:32.011286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.011258    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:00:32.025081 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.025061    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.035550 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.035534    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-64.ec2.internal\" not found" node="ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.036210 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.036192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4547ca9ec435802628cd2ede92cc0a62-config\") pod \"kube-apiserver-proxy-ip-10-0-136-64.ec2.internal\" (UID: \"4547ca9ec435802628cd2ede92cc0a62\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.036292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.036215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9ac5d7847b4bc9b19cbab532fb6eabb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"c9ac5d7847b4bc9b19cbab532fb6eabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.036292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.036234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9ac5d7847b4bc9b19cbab532fb6eabb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"c9ac5d7847b4bc9b19cbab532fb6eabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.039738 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.039722    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-64.ec2.internal\" not found" node="ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.125155 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.125137    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.137259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.137237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9ac5d7847b4bc9b19cbab532fb6eabb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"c9ac5d7847b4bc9b19cbab532fb6eabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.137334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.137268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9ac5d7847b4bc9b19cbab532fb6eabb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"c9ac5d7847b4bc9b19cbab532fb6eabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.137334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.137296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4547ca9ec435802628cd2ede92cc0a62-config\") pod \"kube-apiserver-proxy-ip-10-0-136-64.ec2.internal\" (UID: \"4547ca9ec435802628cd2ede92cc0a62\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.137428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.137347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9ac5d7847b4bc9b19cbab532fb6eabb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"c9ac5d7847b4bc9b19cbab532fb6eabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.137428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.137348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9ac5d7847b4bc9b19cbab532fb6eabb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"c9ac5d7847b4bc9b19cbab532fb6eabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.137428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.137413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4547ca9ec435802628cd2ede92cc0a62-config\") pod \"kube-apiserver-proxy-ip-10-0-136-64.ec2.internal\" (UID: \"4547ca9ec435802628cd2ede92cc0a62\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.226102 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.226075    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.326846 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.326795    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.339136 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.339117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.343475 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.343459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.427725 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.427699    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.528261 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.528233    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.628815 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.628735    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Sep 18 18:00:32.679599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.679568    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:00:32.734729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.734706    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.744172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.744156    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 18:00:32.745678 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.745667    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.754608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.754593    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 18:00:32.754717 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.754700    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 18:00:32.754717 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.754702    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 18:00:32.754794 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.754722    2580 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://aea7b5fd562464ce0a72c4eb199c2370-f4e1f48c00b737a1.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.136.64:38956->34.194.152.253:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Sep 18 18:00:32.788891 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.788872    2580 apiserver.go:52] "Watching apiserver"
Sep 18 18:00:32.804198 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.804177    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 18:00:32.804521 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.804500    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-wppwj","kube-system/global-pull-secret-syncer-qh4kd","kube-system/konnectivity-agent-mchr8","kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b","openshift-image-registry/node-ca-dxk9w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal","openshift-multus/network-metrics-daemon-bs8m5","openshift-network-operator/iptables-alerter-2xcp2","openshift-cluster-node-tuning-operator/tuned-fgjb4","openshift-dns/node-resolver-jjslp","openshift-multus/multus-additional-cni-plugins-74hfr","openshift-multus/multus-s8h5t","openshift-network-diagnostics/network-check-target-lv9rm"]
Sep 18 18:00:32.806945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.806931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.808786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.808765    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 18:00:32.808972 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.808958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 18:00:32.809065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.808985    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.809147 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.809128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.809226 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.809183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.809226 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.809191    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 18:00:32.809226 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.809200    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:32.809331 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.809310    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 18:00:32.809414 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.809399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pb58m\""
Sep 18 18:00:32.811422 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.811404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.813559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.813538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.813891 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.813876    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 18:00:32.813948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.813920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cn946\""
Sep 18 18:00:32.813993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.813963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 18:00:32.815610 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.815595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.816443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.816428    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.816443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.816439    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 18:00:32.816903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.816886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dpl9x\""
Sep 18 18:00:32.816903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.816902    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.817580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.817563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.817802 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.817775    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m8fs6\""
Sep 18 18:00:32.817903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.817798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 18:00:32.817903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.817842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:32.818096 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.817901    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:32.818202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.818189    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.820536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.820522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.822847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.822829    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.822847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.822833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.822987 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.822903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 18:00:32.823242 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.823221    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7lg48\""
Sep 18 18:00:32.823320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.823258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.824909 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.824890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.824992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.824965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5kz7\""
Sep 18 18:00:32.825115 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.825100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.825730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.825712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.827618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.827603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.827618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.827609    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.827729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.827646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rm9dq\""
Sep 18 18:00:32.828336 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.828317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.831286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.830679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 18:00:32.831286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.830942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 18:00:32.831286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.830957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 18:00:32.831286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.831009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nj6s7\""
Sep 18 18:00:32.831286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.831111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 18:00:32.832097 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.832061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.832579 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.832564    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 18:00:32.833983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.833964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 18:00:32.834117 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.834102    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9qxwv\""
Sep 18 18:00:32.834650 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.834630    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 18:00:32.834725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.834681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:32.834766 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.834747    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:32.835970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.835951    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 18:00:32.842075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-lib-modules\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.842159 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e980d25-4957-4cfd-899d-8c31cd4786dd-tmp-dir\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.842159 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.842159 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5l4\" (UniqueName: \"kubernetes.io/projected/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-kube-api-access-cz5l4\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.842159 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-run\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.842296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-cni-netd\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.842296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:32.842296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-daemon-config\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.842296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-slash\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.842296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89e39c3d-f7ce-4828-9c43-faa07512cdf3-iptables-alerter-script\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.842296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-cnibin\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-host\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e980d25-4957-4cfd-899d-8c31cd4786dd-hosts-file\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shsdr\" (UniqueName: \"kubernetes.io/projected/9e980d25-4957-4cfd-899d-8c31cd4786dd-kube-api-access-shsdr\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-cni-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-kubelet\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-env-overrides\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovn-node-metrics-cert\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62d5c61a-8176-416d-874b-b61ae7f28b64-konnectivity-ca\") pod \"konnectivity-agent-mchr8\" (UID: \"62d5c61a-8176-416d-874b-b61ae7f28b64\") " pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rz96\" (UniqueName: \"kubernetes.io/projected/5c8bc962-f106-4454-bbfc-3bffe3b209b6-kube-api-access-9rz96\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-etc-kubernetes\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.842549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-ovn\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffr2\" (UniqueName: \"kubernetes.io/projected/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-kube-api-access-bffr2\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-registration-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-host\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysconfig\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-log-socket\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-cni-bin\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cnibin\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4g7\" (UniqueName: \"kubernetes.io/projected/9033b457-ac6a-487e-935c-418fcafcf450-kube-api-access-rv4g7\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-k8s-cni-cncf-io\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-cni-multus\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-sys-fs\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-system-cni-dir\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.843102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.842977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-serviceca\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-multus-certs\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2ld\" (UniqueName: \"kubernetes.io/projected/4ce9057c-cad7-4604-9e93-b63faa1362ba-kube-api-access-bb2ld\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysctl-conf\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-socket-dir-parent\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-etc-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-etc-selinux\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-cni-bin\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-systemd-units\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-modprobe-d\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-kubernetes\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9033b457-ac6a-487e-935c-418fcafcf450-tmp\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-system-cni-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-kubelet\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.843725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-run-netns\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovnkube-script-lib\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0daa103-d922-4223-bf67-567a3fb989c5-dbus\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89e39c3d-f7ce-4828-9c43-faa07512cdf3-host-slash\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9nb\" (UniqueName: \"kubernetes.io/projected/89e39c3d-f7ce-4828-9c43-faa07512cdf3-kube-api-access-hf9nb\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9033b457-ac6a-487e-935c-418fcafcf450-etc-tuned\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkvs\" (UniqueName: \"kubernetes.io/projected/4ebd8a70-1851-4dc9-89dd-f56715c00883-kube-api-access-svkvs\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovnkube-config\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0daa103-d922-4223-bf67-567a3fb989c5-kubelet-config\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthvt\" (UniqueName: \"kubernetes.io/projected/5b36b266-fa83-480b-8528-43e874c42405-kube-api-access-pthvt\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-systemd\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-conf-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-socket-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-device-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.844344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysctl-d\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-sys\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ebd8a70-1851-4dc9-89dd-f56715c00883-cni-binary-copy\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-netns\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-hostroot\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-systemd\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-var-lib-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-node-log\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-var-lib-kubelet\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.843997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.844769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.844054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62d5c61a-8176-416d-874b-b61ae7f28b64-agent-certs\") pod \"konnectivity-agent-mchr8\" (UID: \"62d5c61a-8176-416d-874b-b61ae7f28b64\") " pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.848688 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.848673    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 18:00:32.870363 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.870310    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 17:55:31 +0000 UTC" deadline="2028-02-25 15:31:35.196449066 +0000 UTC"
Sep 18 18:00:32.870443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.870363    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12597h31m2.326091069s"
Sep 18 18:00:32.910588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.910569    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6kkfd"
Sep 18 18:00:32.916180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.916162    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6kkfd"
Sep 18 18:00:32.944608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-var-lib-kubelet\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.944608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.944723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62d5c61a-8176-416d-874b-b61ae7f28b64-agent-certs\") pod \"konnectivity-agent-mchr8\" (UID: \"62d5c61a-8176-416d-874b-b61ae7f28b64\") " pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.944723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-lib-modules\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.944723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-var-lib-kubelet\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.944723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.944841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e980d25-4957-4cfd-899d-8c31cd4786dd-tmp-dir\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.944841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-lib-modules\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.944841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.944841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5l4\" (UniqueName: \"kubernetes.io/projected/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-kube-api-access-cz5l4\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.944841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-run\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-cni-netd\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-daemon-config\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-slash\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89e39c3d-f7ce-4828-9c43-faa07512cdf3-iptables-alerter-script\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944976    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.945046    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:32.945065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-cnibin\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-cni-netd\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-slash\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-cnibin\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-host\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.945226    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:00:33.445205447 +0000 UTC m=+3.597080277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-host\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e980d25-4957-4cfd-899d-8c31cd4786dd-hosts-file\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shsdr\" (UniqueName: \"kubernetes.io/projected/9e980d25-4957-4cfd-899d-8c31cd4786dd-kube-api-access-shsdr\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-cni-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-kubelet\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-env-overrides\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.945444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovn-node-metrics-cert\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62d5c61a-8176-416d-874b-b61ae7f28b64-konnectivity-ca\") pod \"konnectivity-agent-mchr8\" (UID: \"62d5c61a-8176-416d-874b-b61ae7f28b64\") " pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rz96\" (UniqueName: \"kubernetes.io/projected/5c8bc962-f106-4454-bbfc-3bffe3b209b6-kube-api-access-9rz96\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-etc-kubernetes\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-ovn\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffr2\" (UniqueName: \"kubernetes.io/projected/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-kube-api-access-bffr2\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-registration-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89e39c3d-f7ce-4828-9c43-faa07512cdf3-iptables-alerter-script\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-host\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.944948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-run\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysconfig\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-ovn\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-registration-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-cni-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e980d25-4957-4cfd-899d-8c31cd4786dd-hosts-file\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.946208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-host\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysconfig\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-log-socket\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-cni-bin\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.945983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cnibin\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4g7\" (UniqueName: \"kubernetes.io/projected/9033b457-ac6a-487e-935c-418fcafcf450-kube-api-access-rv4g7\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-k8s-cni-cncf-io\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-cni-multus\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cnibin\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-cni-bin\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-sys-fs\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-env-overrides\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-kubelet\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-system-cni-dir\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-etc-kubernetes\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.948051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62d5c61a-8176-416d-874b-b61ae7f28b64-konnectivity-ca\") pod \"konnectivity-agent-mchr8\" (UID: \"62d5c61a-8176-416d-874b-b61ae7f28b64\") " pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-serviceca\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-multus-certs\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2ld\" (UniqueName: \"kubernetes.io/projected/4ce9057c-cad7-4604-9e93-b63faa1362ba-kube-api-access-bb2ld\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-log-socket\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-k8s-cni-cncf-io\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-sys-fs\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-cni-multus\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-multus-certs\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8bc962-f106-4454-bbfc-3bffe3b209b6-system-cni-dir\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-daemon-config\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysctl-conf\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-socket-dir-parent\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-etc-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-etc-selinux\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-cni-bin\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.949841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-systemd-units\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-etc-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.946976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysctl-conf\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-socket-dir-parent\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-modprobe-d\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-serviceca\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-kubernetes\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-systemd-units\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-var-lib-cni-bin\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-etc-selinux\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9033b457-ac6a-487e-935c-418fcafcf450-tmp\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-modprobe-d\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-system-cni-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-kubelet\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-kubernetes\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.950620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-system-cni-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-run-netns\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-kubelet\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovnkube-script-lib\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-host-run-netns\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0daa103-d922-4223-bf67-567a3fb989c5-dbus\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89e39c3d-f7ce-4828-9c43-faa07512cdf3-host-slash\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0daa103-d922-4223-bf67-567a3fb989c5-dbus\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9nb\" (UniqueName: \"kubernetes.io/projected/89e39c3d-f7ce-4828-9c43-faa07512cdf3-kube-api-access-hf9nb\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9033b457-ac6a-487e-935c-418fcafcf450-etc-tuned\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkvs\" (UniqueName: \"kubernetes.io/projected/4ebd8a70-1851-4dc9-89dd-f56715c00883-kube-api-access-svkvs\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovnkube-config\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.947697    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:32.952372 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.947762    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:33.447733146 +0000 UTC m=+3.599607979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0daa103-d922-4223-bf67-567a3fb989c5-kubelet-config\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8bc962-f106-4454-bbfc-3bffe3b209b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthvt\" (UniqueName: \"kubernetes.io/projected/5b36b266-fa83-480b-8528-43e874c42405-kube-api-access-pthvt\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-systemd\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89e39c3d-f7ce-4828-9c43-faa07512cdf3-host-slash\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0daa103-d922-4223-bf67-567a3fb989c5-kubelet-config\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.947866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-conf-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-socket-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-device-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysctl-d\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-sys\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ebd8a70-1851-4dc9-89dd-f56715c00883-cni-binary-copy\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovnkube-config\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.948800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e980d25-4957-4cfd-899d-8c31cd4786dd-tmp-dir\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-socket-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-netns\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-hostroot\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-systemd\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-var-lib-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-node-log\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62d5c61a-8176-416d-874b-b61ae7f28b64-agent-certs\") pod \"konnectivity-agent-mchr8\" (UID: \"62d5c61a-8176-416d-874b-b61ae7f28b64\") " pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-sys\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9033b457-ac6a-487e-935c-418fcafcf450-tmp\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ebd8a70-1851-4dc9-89dd-f56715c00883-cni-binary-copy\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-multus-conf-dir\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-sysctl-d\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b36b266-fa83-480b-8528-43e874c42405-device-dir\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.949942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovn-node-metrics-cert\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9033b457-ac6a-487e-935c-418fcafcf450-etc-systemd\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-ovnkube-script-lib\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-host-run-netns\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-run-systemd\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-var-lib-openvswitch\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ebd8a70-1851-4dc9-89dd-f56715c00883-hostroot\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.953514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.950237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-node-log\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.954052 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.951448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9033b457-ac6a-487e-935c-418fcafcf450-etc-tuned\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.955512 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.955494    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:00:32.955575 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.955517    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:00:32.955575 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.955527    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:32.955575 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:32.955568    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:33.455558132 +0000 UTC m=+3.607432944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:32.957731 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.956935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5l4\" (UniqueName: \"kubernetes.io/projected/d685ebef-6279-418c-ba6e-dbb5e8b24bd0-kube-api-access-cz5l4\") pod \"node-ca-dxk9w\" (UID: \"d685ebef-6279-418c-ba6e-dbb5e8b24bd0\") " pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:32.958548 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:32.958519    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4547ca9ec435802628cd2ede92cc0a62.slice/crio-a2bac2d345039a0cc610df29d54366ddc04da0bba33ab11a5a2959437ba7be27 WatchSource:0}: Error finding container a2bac2d345039a0cc610df29d54366ddc04da0bba33ab11a5a2959437ba7be27: Status 404 returned error can't find the container with id a2bac2d345039a0cc610df29d54366ddc04da0bba33ab11a5a2959437ba7be27
Sep 18 18:00:32.958888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.958858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2ld\" (UniqueName: \"kubernetes.io/projected/4ce9057c-cad7-4604-9e93-b63faa1362ba-kube-api-access-bb2ld\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:32.958977 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.958956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkvs\" (UniqueName: \"kubernetes.io/projected/4ebd8a70-1851-4dc9-89dd-f56715c00883-kube-api-access-svkvs\") pod \"multus-s8h5t\" (UID: \"4ebd8a70-1851-4dc9-89dd-f56715c00883\") " pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:32.959163 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.959140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rz96\" (UniqueName: \"kubernetes.io/projected/5c8bc962-f106-4454-bbfc-3bffe3b209b6-kube-api-access-9rz96\") pod \"multus-additional-cni-plugins-74hfr\" (UID: \"5c8bc962-f106-4454-bbfc-3bffe3b209b6\") " pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:32.959270 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.959254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthvt\" (UniqueName: \"kubernetes.io/projected/5b36b266-fa83-480b-8528-43e874c42405-kube-api-access-pthvt\") pod \"aws-ebs-csi-driver-node-jqm9b\" (UID: \"5b36b266-fa83-480b-8528-43e874c42405\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:32.959904 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.959871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffr2\" (UniqueName: \"kubernetes.io/projected/08d6e7e1-37e0-42cb-9478-c6eeb211e57e-kube-api-access-bffr2\") pod \"ovnkube-node-wppwj\" (UID: \"08d6e7e1-37e0-42cb-9478-c6eeb211e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:32.959904 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.959890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9nb\" (UniqueName: \"kubernetes.io/projected/89e39c3d-f7ce-4828-9c43-faa07512cdf3-kube-api-access-hf9nb\") pod \"iptables-alerter-2xcp2\" (UID: \"89e39c3d-f7ce-4828-9c43-faa07512cdf3\") " pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:32.960212 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.960195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shsdr\" (UniqueName: \"kubernetes.io/projected/9e980d25-4957-4cfd-899d-8c31cd4786dd-kube-api-access-shsdr\") pod \"node-resolver-jjslp\" (UID: \"9e980d25-4957-4cfd-899d-8c31cd4786dd\") " pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:32.960621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.960601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4g7\" (UniqueName: \"kubernetes.io/projected/9033b457-ac6a-487e-935c-418fcafcf450-kube-api-access-rv4g7\") pod \"tuned-fgjb4\" (UID: \"9033b457-ac6a-487e-935c-418fcafcf450\") " pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:32.964458 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:32.964445    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:00:33.134278 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.134255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:33.142309 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.142270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:33.143599 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.143582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d6e7e1_37e0_42cb_9478_c6eeb211e57e.slice/crio-85ee686ea64610bb62c1084c9af139e5e96f20d73498751d3292a818b80ab9d6 WatchSource:0}: Error finding container 85ee686ea64610bb62c1084c9af139e5e96f20d73498751d3292a818b80ab9d6: Status 404 returned error can't find the container with id 85ee686ea64610bb62c1084c9af139e5e96f20d73498751d3292a818b80ab9d6
Sep 18 18:00:33.152358 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.152337    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d5c61a_8176_416d_874b_b61ae7f28b64.slice/crio-65c9a70c942678fe19649f9006f8d688a1f075238724fdfe95ff446fa36ad291 WatchSource:0}: Error finding container 65c9a70c942678fe19649f9006f8d688a1f075238724fdfe95ff446fa36ad291: Status 404 returned error can't find the container with id 65c9a70c942678fe19649f9006f8d688a1f075238724fdfe95ff446fa36ad291
Sep 18 18:00:33.153091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.153074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b"
Sep 18 18:00:33.157880 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.157864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxk9w"
Sep 18 18:00:33.161182 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.161150    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b36b266_fa83_480b_8528_43e874c42405.slice/crio-639f07acb73c4470140d54edd37a4a32b2e0e0d230d67d4e70e25cc75493be8e WatchSource:0}: Error finding container 639f07acb73c4470140d54edd37a4a32b2e0e0d230d67d4e70e25cc75493be8e: Status 404 returned error can't find the container with id 639f07acb73c4470140d54edd37a4a32b2e0e0d230d67d4e70e25cc75493be8e
Sep 18 18:00:33.172454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.172434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2xcp2"
Sep 18 18:00:33.172691 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.172669    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd685ebef_6279_418c_ba6e_dbb5e8b24bd0.slice/crio-1606cdf3493a8f9e0a9ba79a4488943f846865ffbae7d21c6fa174537144fa1a WatchSource:0}: Error finding container 1606cdf3493a8f9e0a9ba79a4488943f846865ffbae7d21c6fa174537144fa1a: Status 404 returned error can't find the container with id 1606cdf3493a8f9e0a9ba79a4488943f846865ffbae7d21c6fa174537144fa1a
Sep 18 18:00:33.184655 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.184639    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e39c3d_f7ce_4828_9c43_faa07512cdf3.slice/crio-cc10e2e9644333353ffaa0e39d10705c38f49dc62a36be55e513fd9efba45b54 WatchSource:0}: Error finding container cc10e2e9644333353ffaa0e39d10705c38f49dc62a36be55e513fd9efba45b54: Status 404 returned error can't find the container with id cc10e2e9644333353ffaa0e39d10705c38f49dc62a36be55e513fd9efba45b54
Sep 18 18:00:33.185386 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.185366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ac5d7847b4bc9b19cbab532fb6eabb.slice/crio-087c0a29010741de6edca2eba9fa10ed3e72ffdfbb1b9eb1b957ac6a76a49507 WatchSource:0}: Error finding container 087c0a29010741de6edca2eba9fa10ed3e72ffdfbb1b9eb1b957ac6a76a49507: Status 404 returned error can't find the container with id 087c0a29010741de6edca2eba9fa10ed3e72ffdfbb1b9eb1b957ac6a76a49507
Sep 18 18:00:33.189324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.189309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgjb4"
Sep 18 18:00:33.194909 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.194890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jjslp"
Sep 18 18:00:33.197475 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.197452    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9033b457_ac6a_487e_935c_418fcafcf450.slice/crio-0f4cc53f18911f1579c827213d66111253f0ccd03b0fd6d09ab8c27cfac3cde7 WatchSource:0}: Error finding container 0f4cc53f18911f1579c827213d66111253f0ccd03b0fd6d09ab8c27cfac3cde7: Status 404 returned error can't find the container with id 0f4cc53f18911f1579c827213d66111253f0ccd03b0fd6d09ab8c27cfac3cde7
Sep 18 18:00:33.202656 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.202633    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e980d25_4957_4cfd_899d_8c31cd4786dd.slice/crio-e39337f895b8aa0deb44839b7e0dc8a4ecfc885474209c4fb5fa4c70587b64f5 WatchSource:0}: Error finding container e39337f895b8aa0deb44839b7e0dc8a4ecfc885474209c4fb5fa4c70587b64f5: Status 404 returned error can't find the container with id e39337f895b8aa0deb44839b7e0dc8a4ecfc885474209c4fb5fa4c70587b64f5
Sep 18 18:00:33.212513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.212498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74hfr"
Sep 18 18:00:33.218203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.218179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s8h5t"
Sep 18 18:00:33.221242 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.221223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8bc962_f106_4454_bbfc_3bffe3b209b6.slice/crio-3475c32fd9a18f242b45b4432e1f58a9543457ca5acc685c946a84814784654f WatchSource:0}: Error finding container 3475c32fd9a18f242b45b4432e1f58a9543457ca5acc685c946a84814784654f: Status 404 returned error can't find the container with id 3475c32fd9a18f242b45b4432e1f58a9543457ca5acc685c946a84814784654f
Sep 18 18:00:33.223641 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.223625    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:00:33.225834 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:00:33.225814    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebd8a70_1851_4dc9_89dd_f56715c00883.slice/crio-d1ea6027f7605fbab8d9fe3e66808754e76492db14fce3bfe75b1336058af5a8 WatchSource:0}: Error finding container d1ea6027f7605fbab8d9fe3e66808754e76492db14fce3bfe75b1336058af5a8: Status 404 returned error can't find the container with id d1ea6027f7605fbab8d9fe3e66808754e76492db14fce3bfe75b1336058af5a8
Sep 18 18:00:33.309570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.309550    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:00:33.453532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.453461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:33.453532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.453515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:33.453733 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.453658    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:33.453733 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.453713    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:00:34.453695598 +0000 UTC m=+4.605570432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:33.454131 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.454108    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:33.454268 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.454162    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:34.45414675 +0000 UTC m=+4.606021566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:33.554465 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.554435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:33.554645 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.554623    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:00:33.554703 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.554650    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:00:33.554703 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.554664    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:33.554796 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:33.554732    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:34.554704166 +0000 UTC m=+4.706578985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:33.834739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.834657    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:00:33.919106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.919051    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:55:32 +0000 UTC" deadline="2028-04-03 08:41:57.883691991 +0000 UTC"
Sep 18 18:00:33.919106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.919103    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13502h41m23.964592421s"
Sep 18 18:00:33.922407 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.922193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerStarted","Data":"3475c32fd9a18f242b45b4432e1f58a9543457ca5acc685c946a84814784654f"}
Sep 18 18:00:33.928454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.928426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" event={"ID":"c9ac5d7847b4bc9b19cbab532fb6eabb","Type":"ContainerStarted","Data":"087c0a29010741de6edca2eba9fa10ed3e72ffdfbb1b9eb1b957ac6a76a49507"}
Sep 18 18:00:33.931892 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.931864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2xcp2" event={"ID":"89e39c3d-f7ce-4828-9c43-faa07512cdf3","Type":"ContainerStarted","Data":"cc10e2e9644333353ffaa0e39d10705c38f49dc62a36be55e513fd9efba45b54"}
Sep 18 18:00:33.947311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.947262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxk9w" event={"ID":"d685ebef-6279-418c-ba6e-dbb5e8b24bd0","Type":"ContainerStarted","Data":"1606cdf3493a8f9e0a9ba79a4488943f846865ffbae7d21c6fa174537144fa1a"}
Sep 18 18:00:33.949138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.949105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"85ee686ea64610bb62c1084c9af139e5e96f20d73498751d3292a818b80ab9d6"}
Sep 18 18:00:33.951900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.951823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s8h5t" event={"ID":"4ebd8a70-1851-4dc9-89dd-f56715c00883","Type":"ContainerStarted","Data":"d1ea6027f7605fbab8d9fe3e66808754e76492db14fce3bfe75b1336058af5a8"}
Sep 18 18:00:33.958483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.958243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jjslp" event={"ID":"9e980d25-4957-4cfd-899d-8c31cd4786dd","Type":"ContainerStarted","Data":"e39337f895b8aa0deb44839b7e0dc8a4ecfc885474209c4fb5fa4c70587b64f5"}
Sep 18 18:00:33.966983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.966959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgjb4" event={"ID":"9033b457-ac6a-487e-935c-418fcafcf450","Type":"ContainerStarted","Data":"0f4cc53f18911f1579c827213d66111253f0ccd03b0fd6d09ab8c27cfac3cde7"}
Sep 18 18:00:33.971158 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.971134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b" event={"ID":"5b36b266-fa83-480b-8528-43e874c42405","Type":"ContainerStarted","Data":"639f07acb73c4470140d54edd37a4a32b2e0e0d230d67d4e70e25cc75493be8e"}
Sep 18 18:00:33.973726 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.973685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mchr8" event={"ID":"62d5c61a-8176-416d-874b-b61ae7f28b64","Type":"ContainerStarted","Data":"65c9a70c942678fe19649f9006f8d688a1f075238724fdfe95ff446fa36ad291"}
Sep 18 18:00:33.979122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:33.979099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal" event={"ID":"4547ca9ec435802628cd2ede92cc0a62","Type":"ContainerStarted","Data":"a2bac2d345039a0cc610df29d54366ddc04da0bba33ab11a5a2959437ba7be27"}
Sep 18 18:00:34.463240 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.463203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:34.463414 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.463287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:34.463414 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.463406    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:34.463531 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.463464    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:36.463448255 +0000 UTC m=+6.615323082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:34.463900 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.463882    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:34.463963 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.463938    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:00:36.463922907 +0000 UTC m=+6.615797723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:34.564180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.564118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:34.564367 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.564283    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:00:34.564367 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.564305    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:00:34.564367 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.564318    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:34.564514 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.564376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:36.564358379 +0000 UTC m=+6.716233210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:34.827837 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.827765    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:00:34.904689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.903758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:34.904689 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.903884    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:34.904689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.904267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:34.904689 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.904353    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:34.904689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.904411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:34.904689 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:34.904478    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:34.920234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.920188    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:55:32 +0000 UTC" deadline="2028-05-25 20:17:14.601177882 +0000 UTC"
Sep 18 18:00:34.920234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:34.920219    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14762h16m39.680963278s"
Sep 18 18:00:36.481749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:36.481709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:36.482239 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:36.481802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:36.482239 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.481917    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:36.482239 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.481987    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:40.481967667 +0000 UTC m=+10.633842495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:36.482430 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.482379    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:36.482430 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.482429    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:00:40.482413545 +0000 UTC m=+10.634288376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:36.582681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:36.582648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:36.582844 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.582822    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:00:36.582922 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.582848    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:00:36.582922 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.582861    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:36.582922 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.582917    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:40.582898295 +0000 UTC m=+10.734773111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:36.904236 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:36.904160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:36.904389 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.904307    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:36.904685 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:36.904663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:36.904762 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.904737    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:36.904820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:36.904768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:36.904820 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:36.904809    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:38.903964 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:38.903930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:38.904437 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:38.904066    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:38.904437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:38.904419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:38.904551 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:38.904518    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:38.904613 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:38.904564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:38.909734 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:38.904993    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:40.519430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:40.518928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:40.519430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:40.518970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:40.519430 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.519100    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:40.519430 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.519101    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:40.519430 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.519156    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:00:48.519140405 +0000 UTC m=+18.671015225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:40.519430 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.519172    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:48.519164115 +0000 UTC m=+18.671038929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:40.619832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:40.619669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:40.620563 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.620146    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:00:40.620563 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.620174    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:00:40.620563 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.620186    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:40.620563 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.620245    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:48.620225455 +0000 UTC m=+18.772100271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:40.903961 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:40.903867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:40.904143 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.904001    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:40.904143 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:40.904093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:40.904255 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.904176    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:40.904255 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:40.904226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:40.904398 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:40.904292    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:42.904538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:42.904067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:42.904538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:42.904096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:42.904538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:42.904112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:42.904538 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:42.904210    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:42.904538 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:42.904355    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:42.904538 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:42.904423    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:44.904586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:44.904554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:44.904586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:44.904583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:44.905113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:44.904591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:44.905113 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:44.904683    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:44.905113 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:44.904808    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:44.905113 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:44.904881    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:46.903806 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:46.903770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:46.904249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:46.903811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:46.904249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:46.903899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:46.904249 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:46.903915    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:46.904249 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:46.904005    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:46.904249 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:46.904113    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:48.578961 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:48.578925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:48.579504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:48.579005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:48.579504 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.579111    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:48.579504 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.579118    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:48.579504 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.579176    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:01:04.579159424 +0000 UTC m=+34.731034250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:00:48.579504 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.579196    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:04.579185879 +0000 UTC m=+34.731060695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:00:48.680250 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:48.680213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:48.680429 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.680404    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:00:48.680429 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.680428    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:00:48.680537 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.680441    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:48.680537 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.680501    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:04.680482572 +0000 UTC m=+34.832357399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:00:48.904384 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:48.904310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:48.904384 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:48.904330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:48.904580 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.904426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:48.904580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:48.904434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:48.904580 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.904495    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:48.904580 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:48.904568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:50.904273 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:50.904241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:50.904737 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:50.904251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:50.904737 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:50.904397    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:50.904737 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:50.904251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:50.904737 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:50.904479    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:50.904737 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:50.904519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:52.022598 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.022416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal" event={"ID":"4547ca9ec435802628cd2ede92cc0a62","Type":"ContainerStarted","Data":"c725bcfc640ad3739741c13540f2f9dde290204959b7d74dbc0f6ea1e6421721"}
Sep 18 18:00:52.025093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.025063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"3bb556ec4ecf813046f7e4893d0d5a0e952123759dbc396472efda54c423eb6f"}
Sep 18 18:00:52.028782 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.028604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s8h5t" event={"ID":"4ebd8a70-1851-4dc9-89dd-f56715c00883","Type":"ContainerStarted","Data":"cb4c3c88880b1848db56e3ebda83a348e5aa6f51149ab26606e96f1ef034e9e3"}
Sep 18 18:00:52.034463 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.034400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgjb4" event={"ID":"9033b457-ac6a-487e-935c-418fcafcf450","Type":"ContainerStarted","Data":"c588d1fe73307ed07c33306e1e5f18601842ea07ab25ad2597319760706802b8"}
Sep 18 18:00:52.040673 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.040555    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal" podStartSLOduration=20.040536923 podStartE2EDuration="20.040536923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:52.039799372 +0000 UTC m=+22.191674208" watchObservedRunningTime="2026-09-18 18:00:52.040536923 +0000 UTC m=+22.192411758"
Sep 18 18:00:52.058250 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.058198    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s8h5t" podStartSLOduration=1.620154788 podStartE2EDuration="20.058183688s" totalImagesPullingTime="18.4380289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:52.057486269 +0000 UTC m=+22.209361106" watchObservedRunningTime="2026-09-18 18:00:52.058183688 +0000 UTC m=+22.210058526"
Sep 18 18:00:52.072587 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.072513    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fgjb4" podStartSLOduration=1.64269922 podStartE2EDuration="20.072494467s" totalImagesPullingTime="18.429795247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:52.072046322 +0000 UTC m=+22.223921148" watchObservedRunningTime="2026-09-18 18:00:52.072494467 +0000 UTC m=+22.224369303"
Sep 18 18:00:52.904068 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.903965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:52.904219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.903965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:52.904219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:52.903965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:52.904219 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:52.904141    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:52.904321 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:52.904219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:52.904321 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:52.904284    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:53.039246 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.038870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jjslp" event={"ID":"9e980d25-4957-4cfd-899d-8c31cd4786dd","Type":"ContainerStarted","Data":"30f189c2c1652ca20f2c3374852ff5c96914bf6c766cda0ba43b039d1979f3d7"}
Sep 18 18:00:53.040632 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.040606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b" event={"ID":"5b36b266-fa83-480b-8528-43e874c42405","Type":"ContainerStarted","Data":"f265212fe07e4198d4fede49fdb9b604880c248e134af0552d7ebec82bfc9b78"}
Sep 18 18:00:53.041988 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.041966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mchr8" event={"ID":"62d5c61a-8176-416d-874b-b61ae7f28b64","Type":"ContainerStarted","Data":"362d69ce92d79bf8edf88c188655f5a61ef7df9b589aac28d872c91702dae171"}
Sep 18 18:00:53.043662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.043642    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c8bc962-f106-4454-bbfc-3bffe3b209b6" containerID="c5776e4dab0ac14f32bcda24f2e289becce4c6a0715f35fb08ac478687f8bcea" exitCode=0
Sep 18 18:00:53.043741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.043704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerDied","Data":"c5776e4dab0ac14f32bcda24f2e289becce4c6a0715f35fb08ac478687f8bcea"}
Sep 18 18:00:53.045526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.045504    2580 generic.go:356] "Generic (PLEG): container finished" podID="c9ac5d7847b4bc9b19cbab532fb6eabb" containerID="98856cdfbbee64a53831c4e6e904025d6bdd524557a2915a2d8449b1e6cc3540" exitCode=0
Sep 18 18:00:53.045606 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.045582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" event={"ID":"c9ac5d7847b4bc9b19cbab532fb6eabb","Type":"ContainerDied","Data":"98856cdfbbee64a53831c4e6e904025d6bdd524557a2915a2d8449b1e6cc3540"}
Sep 18 18:00:53.048734 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.048713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxk9w" event={"ID":"d685ebef-6279-418c-ba6e-dbb5e8b24bd0","Type":"ContainerStarted","Data":"00b70e31f62ff5ff742b56aae8c41cf4136f115a3e621301c33e41b6be2ae2fa"}
Sep 18 18:00:53.052064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.052044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"2236f4ca267ee04e0778a82cca233af6a3d5307689097170f4921f073becf75f"}
Sep 18 18:00:53.052149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.052118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"29be3dd67a4aa7545d3d238c298c33a58a2fab5f5ea8324b9e23dc27acbac621"}
Sep 18 18:00:53.052207 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.052177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"7fdaa3b9e90c267b501c0a42fb37fbaae617896cfe370ee64be2074f2b74df9f"}
Sep 18 18:00:53.052262 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.052228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"928489202e98f2eba2103dd9bae46897ef79bc2a3c82b8267ce02d35d089536d"}
Sep 18 18:00:53.052311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.052277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"970bfb859e00d7964494fac6eb16a4ecc503f5ce3d7391bb1ef2fc25ae71807b"}
Sep 18 18:00:53.054091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.054044    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jjslp" podStartSLOduration=2.628387986 podStartE2EDuration="21.054008613s" totalImagesPullingTime="18.425620627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:53.053665603 +0000 UTC m=+23.205540449" watchObservedRunningTime="2026-09-18 18:00:53.054008613 +0000 UTC m=+23.205883450"
Sep 18 18:00:53.068209 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.068188    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 18:00:53.081396 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.081347    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mchr8" podStartSLOduration=2.832703252 podStartE2EDuration="21.081329038s" totalImagesPullingTime="18.248625786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:53.080679041 +0000 UTC m=+23.232553877" watchObservedRunningTime="2026-09-18 18:00:53.081329038 +0000 UTC m=+23.233203873"
Sep 18 18:00:53.114259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.114156    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dxk9w" podStartSLOduration=2.6628858920000003 podStartE2EDuration="21.114144382s" totalImagesPullingTime="18.45125849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:53.113790869 +0000 UTC m=+23.265665704" watchObservedRunningTime="2026-09-18 18:00:53.114144382 +0000 UTC m=+23.266019216"
Sep 18 18:00:53.129338 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.129301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2xcp2" podStartSLOduration=10.981394022 podStartE2EDuration="21.129288954s" totalImagesPullingTime="10.147894932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:53.129275421 +0000 UTC m=+23.281150256" watchObservedRunningTime="2026-09-18 18:00:53.129288954 +0000 UTC m=+23.281163776"
Sep 18 18:00:53.189232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.189195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:53.264135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.264102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:53.332947 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.332920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:53.916385 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.916304    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T18:00:53.068205451Z","UUID":"ef0de6fd-be59-4ab7-9e5b-6fa08cf69a5c","Handler":null,"Name":"","Endpoint":""}
Sep 18 18:00:53.918003 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.917981    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 18:00:53.918238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:53.918020    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 18:00:54.056753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.056722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b" event={"ID":"5b36b266-fa83-480b-8528-43e874c42405","Type":"ContainerStarted","Data":"0da2e40b2c96273f0b1e9b8d5e5035a35d09cd32aef5778ec2eda08a5e6522c4"}
Sep 18 18:00:54.057195 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.056793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b" event={"ID":"5b36b266-fa83-480b-8528-43e874c42405","Type":"ContainerStarted","Data":"4130a35e21452bd1d2b581eea287bbc587e6a3e8ee41d655f2818f221cf67f76"}
Sep 18 18:00:54.059178 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.059118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" event={"ID":"c9ac5d7847b4bc9b19cbab532fb6eabb","Type":"ContainerStarted","Data":"68ab8b0a46037b4bae90e32525c8e92cb2f9165ec5659a7e029b9b91b0fc2263"}
Sep 18 18:00:54.060810 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.060781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2xcp2" event={"ID":"89e39c3d-f7ce-4828-9c43-faa07512cdf3","Type":"ContainerStarted","Data":"915dd74a90949c5d1f96420efc1d39b7afd5ca71dc2130138bffac121799cbbe"}
Sep 18 18:00:54.073656 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.073612    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqm9b" podStartSLOduration=2.3241280939999998 podStartE2EDuration="22.073594811s" totalImagesPullingTime="19.749466717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:54.073070567 +0000 UTC m=+24.224945401" watchObservedRunningTime="2026-09-18 18:00:54.073594811 +0000 UTC m=+24.225469651"
Sep 18 18:00:54.087011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.086978    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" podStartSLOduration=22.086966609 podStartE2EDuration="22.086966609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:54.086570548 +0000 UTC m=+24.238445383" watchObservedRunningTime="2026-09-18 18:00:54.086966609 +0000 UTC m=+24.238841443"
Sep 18 18:00:54.097571 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.097544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:54.479246 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.479222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:54.580234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.580200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:54.895560 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.895481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:54.904419 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.904392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:54.904549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.904474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:54.904549 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:54.904480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:54.904549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:54.904485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:54.904659 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:54.904550    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:54.904659 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:54.904619    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:55.067169 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:55.067133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"ac475d14bcc5548925892ec8695435f3e8cfb14c9dd61e22fc713eaf1eb36ad3"}
Sep 18 18:00:56.903972 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:56.903880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:56.903972 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:56.903930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:56.904404 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:56.903989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:56.904404 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:56.903996    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:56.904404 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:56.904091    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:56.904404 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:56.904187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:57.705920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:57.705884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:57.706536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:57.706507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:58.076621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.076580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" event={"ID":"08d6e7e1-37e0-42cb-9478-c6eeb211e57e","Type":"ContainerStarted","Data":"90bb164676486c721ee2f3de5b98cbc1c56f9ce8b40cbb911a5e41599a7368c6"}
Sep 18 18:00:58.076971 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.076926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:58.077327 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.077311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mchr8"
Sep 18 18:00:58.104423 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.104283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj" podStartSLOduration=7.399079719 podStartE2EDuration="26.104268926s" totalImagesPullingTime="18.705189207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:58.103292295 +0000 UTC m=+28.255167130" watchObservedRunningTime="2026-09-18 18:00:58.104268926 +0000 UTC m=+28.256143761"
Sep 18 18:00:58.170118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.170075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:58.270586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.270527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:58.327016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.326992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:58.903904 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.903873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:58.903904 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.903873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:58.904208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:58.903895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:58.904208 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:58.903972    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:58.904208 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:58.904062    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:00:58.904208 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:58.904130    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:59.080488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.080455    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c8bc962-f106-4454-bbfc-3bffe3b209b6" containerID="117d4bb34b9c0b1faf43010c9802fe1f37f4c9a9b235aed0af13b652cd96e8b8" exitCode=0
Sep 18 18:00:59.081108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.080545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerDied","Data":"117d4bb34b9c0b1faf43010c9802fe1f37f4c9a9b235aed0af13b652cd96e8b8"}
Sep 18 18:00:59.081108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.080716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:59.081340 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.081308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:59.081460 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.081378    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:59.096420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.096398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:59.096584 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.096564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:00:59.880624 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.880597    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qh4kd"]
Sep 18 18:00:59.880780 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.880698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:00:59.880842 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:59.880782    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:00:59.882091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.882017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lv9rm"]
Sep 18 18:00:59.882203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.882152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:00:59.882263 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:59.882244    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:00:59.882772 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.882693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bs8m5"]
Sep 18 18:00:59.882869 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:00:59.882791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:00:59.882925 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:00:59.882892    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:01:00.084643 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:00.084421    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c8bc962-f106-4454-bbfc-3bffe3b209b6" containerID="f4fb35349514b25fdf43aa4a1d2622d6774cef51a42a140d8a23f4166dcdd92d" exitCode=0
Sep 18 18:01:00.085213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:00.084508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerDied","Data":"f4fb35349514b25fdf43aa4a1d2622d6774cef51a42a140d8a23f4166dcdd92d"}
Sep 18 18:01:00.904444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:00.904390    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:00.904551 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:00.904496    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:01:01.089457 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:01.089429    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c8bc962-f106-4454-bbfc-3bffe3b209b6" containerID="5363be0474576e5e91559e747af667d6ba9138c46636e77354662fae47ecf63d" exitCode=0
Sep 18 18:01:01.089840 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:01.089522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerDied","Data":"5363be0474576e5e91559e747af667d6ba9138c46636e77354662fae47ecf63d"}
Sep 18 18:01:01.467488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:01.467456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:01.523905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:01.523882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:01.907474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:01.907445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:01.907596 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:01.907560    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:01:01.907644 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:01.907623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:01.907749 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:01.907729    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:01:02.101853 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:02.101820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:02.202753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:02.202669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:02.904583 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:02.904558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:02.904752 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:02.904652    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:01:03.904201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:03.904167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:03.904652 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:03.904282    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lv9rm" podUID="a79643fb-2296-47a6-9f3a-0246a0f76039"
Sep 18 18:01:03.904652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:03.904337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:03.904652 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:03.904419    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qh4kd" podUID="d0daa103-d922-4223-bf67-567a3fb989c5"
Sep 18 18:01:04.342573 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.342525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:04.416164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.416053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:04.606277 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.606198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:04.606277 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.606276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:04.606465 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.606356    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:01:04.606465 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.606403    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:01:04.606465 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.606422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs podName:4ce9057c-cad7-4604-9e93-b63faa1362ba nodeName:}" failed. No retries permitted until 2026-09-18 18:01:36.606408318 +0000 UTC m=+66.758283152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs") pod "network-metrics-daemon-bs8m5" (UID: "4ce9057c-cad7-4604-9e93-b63faa1362ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:01:04.606465 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.606444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret podName:d0daa103-d922-4223-bf67-567a3fb989c5 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:36.606431759 +0000 UTC m=+66.758306572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret") pod "global-pull-secret-syncer-qh4kd" (UID: "d0daa103-d922-4223-bf67-567a3fb989c5") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:01:04.706933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.706908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:04.707105 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.707062    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:01:04.707105 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.707076    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:01:04.707105 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.707085    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wb89t for pod openshift-network-diagnostics/network-check-target-lv9rm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:01:04.707226 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.707140    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t podName:a79643fb-2296-47a6-9f3a-0246a0f76039 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:36.707121848 +0000 UTC m=+66.858996666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wb89t" (UniqueName: "kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t") pod "network-check-target-lv9rm" (UID: "a79643fb-2296-47a6-9f3a-0246a0f76039") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:01:04.718835 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.718759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:04.813989 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.813883    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:04.904727 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.904661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:04.905128 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:04.904770    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs8m5" podUID="4ce9057c-cad7-4604-9e93-b63faa1362ba"
Sep 18 18:01:04.912640 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:04.912618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:05.014639 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.014609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:05.523686 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.523654    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jjslp_9e980d25-4957-4cfd-899d-8c31cd4786dd/dns-node-resolver/0.log"
Sep 18 18:01:05.831441 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.831419    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeReady"
Sep 18 18:01:05.831560 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.831544    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 18:01:05.883383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.883357    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jnwcp"]
Sep 18 18:01:05.886392 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.886373    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fmzmg"]
Sep 18 18:01:05.886501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.886485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:05.888628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.888454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 18:01:05.888628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.888564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhsq2\""
Sep 18 18:01:05.888797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.888636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 18:01:05.889585 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.889520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:05.890226 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.890192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:05.892038 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.891932    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 18:01:05.892038 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.891977    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 18:01:05.892038 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.892018    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 18:01:05.892258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.892150    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wnjwf\""
Sep 18 18:01:05.899696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.899676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmzmg"]
Sep 18 18:01:05.900566 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.900543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnwcp"]
Sep 18 18:01:05.907907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.907885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:05.908232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.908094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:05.913072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.912643    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 18:01:05.913072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.912673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fqhj6\""
Sep 18 18:01:05.913072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.912709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 18:01:05.913072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.912950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 18:01:05.913901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.913882    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wtwbq"]
Sep 18 18:01:05.917490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.917471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:05.920411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.920391    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 18:01:05.920492 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.920391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 18:01:05.920492 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.920394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dd246\""
Sep 18 18:01:05.920492 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.920473    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 18:01:05.920723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.920707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 18:01:05.927186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.927167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wtwbq"]
Sep 18 18:01:05.965959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:05.965922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:06.015839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.015818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtt9\" (UniqueName: \"kubernetes.io/projected/b65eb7ba-3c83-4ff3-acee-793412b227c5-kube-api-access-gwtt9\") pod \"ingress-canary-fmzmg\" (UID: \"b65eb7ba-3c83-4ff3-acee-793412b227c5\") " pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.015906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.015871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57d4563-a4fe-4512-92de-b33ab932fc1b-data-volume\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.015906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.015895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.015966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.015931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542cb77a-84d0-433d-9d8f-36cb17a660f2-tmp-dir\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.015998 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.015973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5x4j\" (UniqueName: \"kubernetes.io/projected/542cb77a-84d0-433d-9d8f-36cb17a660f2-kube-api-access-l5x4j\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.016087 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.016015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.016121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.016104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/542cb77a-84d0-433d-9d8f-36cb17a660f2-config-volume\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.016153 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.016131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.016153 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.016147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b65eb7ba-3c83-4ff3-acee-793412b227c5-cert\") pod \"ingress-canary-fmzmg\" (UID: \"b65eb7ba-3c83-4ff3-acee-793412b227c5\") " pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.016214 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.016190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a57d4563-a4fe-4512-92de-b33ab932fc1b-crio-socket\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.016243 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.016212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tlg\" (UniqueName: \"kubernetes.io/projected/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-api-access-q4tlg\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.066553 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.066494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:06.104008 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.103981    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c8bc962-f106-4454-bbfc-3bffe3b209b6" containerID="02239da78fb864b12f2f8f636046f6a5b17e0504a2b388328ebd9a94c962bdfc" exitCode=0
Sep 18 18:01:06.104114 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.104049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerDied","Data":"02239da78fb864b12f2f8f636046f6a5b17e0504a2b388328ebd9a94c962bdfc"}
Sep 18 18:01:06.117164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57d4563-a4fe-4512-92de-b33ab932fc1b-data-volume\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.117255 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.117255 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542cb77a-84d0-433d-9d8f-36cb17a660f2-tmp-dir\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.117255 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5x4j\" (UniqueName: \"kubernetes.io/projected/542cb77a-84d0-433d-9d8f-36cb17a660f2-kube-api-access-l5x4j\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.117398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.117398 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.117278    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:01:06.117398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/542cb77a-84d0-433d-9d8f-36cb17a660f2-config-volume\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.117398 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.117341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:06.617322621 +0000 UTC m=+36.769197435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : configmap "kube-rbac-proxy" not found
Sep 18 18:01:06.117398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b65eb7ba-3c83-4ff3-acee-793412b227c5-cert\") pod \"ingress-canary-fmzmg\" (UID: \"b65eb7ba-3c83-4ff3-acee-793412b227c5\") " pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a57d4563-a4fe-4512-92de-b33ab932fc1b-crio-socket\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tlg\" (UniqueName: \"kubernetes.io/projected/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-api-access-q4tlg\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtt9\" (UniqueName: \"kubernetes.io/projected/b65eb7ba-3c83-4ff3-acee-793412b227c5-kube-api-access-gwtt9\") pod \"ingress-canary-fmzmg\" (UID: \"b65eb7ba-3c83-4ff3-acee-793412b227c5\") " pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57d4563-a4fe-4512-92de-b33ab932fc1b-data-volume\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542cb77a-84d0-433d-9d8f-36cb17a660f2-tmp-dir\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.117609    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:01:06.117647 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.117652    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls podName:542cb77a-84d0-433d-9d8f-36cb17a660f2 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:06.617636091 +0000 UTC m=+36.769510912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls") pod "dns-default-jnwcp" (UID: "542cb77a-84d0-433d-9d8f-36cb17a660f2") : secret "dns-default-metrics-tls" not found
Sep 18 18:01:06.118042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a57d4563-a4fe-4512-92de-b33ab932fc1b-crio-socket\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.118042 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.117812    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:06.118042 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.117855    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:06.617841356 +0000 UTC m=+36.769716172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:06.118042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.117934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/542cb77a-84d0-433d-9d8f-36cb17a660f2-config-volume\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.122094 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.122054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b65eb7ba-3c83-4ff3-acee-793412b227c5-cert\") pod \"ingress-canary-fmzmg\" (UID: \"b65eb7ba-3c83-4ff3-acee-793412b227c5\") " pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.126125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.126103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtt9\" (UniqueName: \"kubernetes.io/projected/b65eb7ba-3c83-4ff3-acee-793412b227c5-kube-api-access-gwtt9\") pod \"ingress-canary-fmzmg\" (UID: \"b65eb7ba-3c83-4ff3-acee-793412b227c5\") " pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.129047 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.129007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tlg\" (UniqueName: \"kubernetes.io/projected/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-api-access-q4tlg\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.129605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.129586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5x4j\" (UniqueName: \"kubernetes.io/projected/542cb77a-84d0-433d-9d8f-36cb17a660f2-kube-api-access-l5x4j\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.204077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.204053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmzmg"
Sep 18 18:01:06.308234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.308203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:06.374398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.374334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:06.474541 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.474508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:06.525488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.525467    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxk9w_d685ebef-6279-418c-ba6e-dbb5e8b24bd0/node-ca/0.log"
Sep 18 18:01:06.525759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.525737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmzmg"]
Sep 18 18:01:06.529126 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:06.529103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65eb7ba_3c83_4ff3_acee_793412b227c5.slice/crio-87fe1865efe66c7899953638247dd92064bb7ecd37bf350d0bc519bb92950b5a WatchSource:0}: Error finding container 87fe1865efe66c7899953638247dd92064bb7ecd37bf350d0bc519bb92950b5a: Status 404 returned error can't find the container with id 87fe1865efe66c7899953638247dd92064bb7ecd37bf350d0bc519bb92950b5a
Sep 18 18:01:06.622238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.622220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:06.622334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.622260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.622334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.622318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:06.622405 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.622363    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:01:06.622405 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.622378    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:01:06.622461 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.622417    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls podName:542cb77a-84d0-433d-9d8f-36cb17a660f2 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:07.622402545 +0000 UTC m=+37.774277376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls") pod "dns-default-jnwcp" (UID: "542cb77a-84d0-433d-9d8f-36cb17a660f2") : secret "dns-default-metrics-tls" not found
Sep 18 18:01:06.622461 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.622433    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:06.622461 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.622436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:07.622426295 +0000 UTC m=+37.774301122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : configmap "kube-rbac-proxy" not found
Sep 18 18:01:06.622553 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:06.622474    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:07.622461212 +0000 UTC m=+37.774336025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:06.904002 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.903977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:06.905981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.905950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 18:01:06.906113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:06.906053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxcq\""
Sep 18 18:01:07.110909 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.110831    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c8bc962-f106-4454-bbfc-3bffe3b209b6" containerID="cee64d44486647fbee86f6799b276adf7f91be03d46d29313d7476515ee23fba" exitCode=0
Sep 18 18:01:07.111731 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.110925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerDied","Data":"cee64d44486647fbee86f6799b276adf7f91be03d46d29313d7476515ee23fba"}
Sep 18 18:01:07.112477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.112452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmzmg" event={"ID":"b65eb7ba-3c83-4ff3-acee-793412b227c5","Type":"ContainerStarted","Data":"87fe1865efe66c7899953638247dd92064bb7ecd37bf350d0bc519bb92950b5a"}
Sep 18 18:01:07.179784 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.179756    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gmztj"]
Sep 18 18:01:07.213230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.213203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:07.216847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.216828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gmztj"]
Sep 18 18:01:07.216949 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.216907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.219888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.219290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 18:01:07.219888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.219296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 18:01:07.219888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.219313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 18:01:07.219888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.219501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fftlz\""
Sep 18 18:01:07.219888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.219745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 18:01:07.313876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.313691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:07.326747 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.326722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29cf4622-cb11-4106-9881-7c2d92d89f3d-signing-key\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.326891 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.326790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn994\" (UniqueName: \"kubernetes.io/projected/29cf4622-cb11-4106-9881-7c2d92d89f3d-kube-api-access-tn994\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.327002 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.326923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29cf4622-cb11-4106-9881-7c2d92d89f3d-signing-cabundle\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.428074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.428021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29cf4622-cb11-4106-9881-7c2d92d89f3d-signing-cabundle\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.428232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.428127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29cf4622-cb11-4106-9881-7c2d92d89f3d-signing-key\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.428232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.428184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn994\" (UniqueName: \"kubernetes.io/projected/29cf4622-cb11-4106-9881-7c2d92d89f3d-kube-api-access-tn994\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.428758 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.428730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29cf4622-cb11-4106-9881-7c2d92d89f3d-signing-cabundle\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.431551 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.431518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29cf4622-cb11-4106-9881-7c2d92d89f3d-signing-key\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.437596 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.437574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn994\" (UniqueName: \"kubernetes.io/projected/29cf4622-cb11-4106-9881-7c2d92d89f3d-kube-api-access-tn994\") pod \"service-ca-55f6fc7b86-gmztj\" (UID: \"29cf4622-cb11-4106-9881-7c2d92d89f3d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.539659 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.539622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj"
Sep 18 18:01:07.629599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.629566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:07.629715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.629674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:07.629715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.629698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:07.629801 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:07.629723    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:07.629801 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:07.629797    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:09.629778353 +0000 UTC m=+39.781653194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:07.629801 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:07.629796    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:01:07.629947 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:07.629806    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:01:07.629947 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:07.629847    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls podName:542cb77a-84d0-433d-9d8f-36cb17a660f2 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:09.629832711 +0000 UTC m=+39.781707536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls") pod "dns-default-jnwcp" (UID: "542cb77a-84d0-433d-9d8f-36cb17a660f2") : secret "dns-default-metrics-tls" not found
Sep 18 18:01:07.629947 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:07.629865    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:09.629854837 +0000 UTC m=+39.781729650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : configmap "kube-rbac-proxy" not found
Sep 18 18:01:07.829214 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:07.829100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gmztj"]
Sep 18 18:01:07.834086 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:07.834059    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cf4622_cb11_4106_9881_7c2d92d89f3d.slice/crio-931dccabd0d13b1242b874de78af1a74274b629ddf71aeb054d7180ff3e77b37 WatchSource:0}: Error finding container 931dccabd0d13b1242b874de78af1a74274b629ddf71aeb054d7180ff3e77b37: Status 404 returned error can't find the container with id 931dccabd0d13b1242b874de78af1a74274b629ddf71aeb054d7180ff3e77b37
Sep 18 18:01:08.116058 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:08.115970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj" event={"ID":"29cf4622-cb11-4106-9881-7c2d92d89f3d","Type":"ContainerStarted","Data":"931dccabd0d13b1242b874de78af1a74274b629ddf71aeb054d7180ff3e77b37"}
Sep 18 18:01:08.119915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:08.119887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74hfr" event={"ID":"5c8bc962-f106-4454-bbfc-3bffe3b209b6","Type":"ContainerStarted","Data":"aa76a56fd23b5bd8173ff034745fca7dd125ebf451bf8f249810cc8b6382d967"}
Sep 18 18:01:08.147641 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:08.147588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-74hfr" podStartSLOduration=13.79166702 podStartE2EDuration="36.147573567s" totalImagesPullingTime="22.355906547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:08.145928916 +0000 UTC m=+38.297803760" watchObservedRunningTime="2026-09-18 18:01:08.147573567 +0000 UTC m=+38.299448393"
Sep 18 18:01:08.283294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:08.283248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:08.671369 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:08.671329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:09.495162 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.495124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:09.555121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.555039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:09.645301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.645258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:09.645301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.645299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:09.645498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.645344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:09.645498 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:09.645477    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:09.645498 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:09.645477    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:01:09.645614 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:09.645545    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls podName:542cb77a-84d0-433d-9d8f-36cb17a660f2 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:13.645525479 +0000 UTC m=+43.797400293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls") pod "dns-default-jnwcp" (UID: "542cb77a-84d0-433d-9d8f-36cb17a660f2") : secret "dns-default-metrics-tls" not found
Sep 18 18:01:09.645614 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:09.645562    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:13.645553506 +0000 UTC m=+43.797428325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:09.645795 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.645778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a57d4563-a4fe-4512-92de-b33ab932fc1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:09.770454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.770418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:09.867844 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:09.867760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:10.126432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:10.126356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmzmg" event={"ID":"b65eb7ba-3c83-4ff3-acee-793412b227c5","Type":"ContainerStarted","Data":"df40928ceef7f0cb4e4cbd98cd099da8caeb3c004ee1852efb450a8fd469bf1e"}
Sep 18 18:01:10.141821 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:10.141775    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fmzmg" podStartSLOduration=2.421166478 podStartE2EDuration="5.141760925s" totalImagesPullingTime="2.720594447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:10.141002996 +0000 UTC m=+40.292877833" watchObservedRunningTime="2026-09-18 18:01:10.141760925 +0000 UTC m=+40.293635769"
Sep 18 18:01:11.656519 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:11.656489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:11.994792 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:11.994755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:12.133086 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.133049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:12.135705 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.135669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj" event={"ID":"29cf4622-cb11-4106-9881-7c2d92d89f3d","Type":"ContainerStarted","Data":"2339ab5294869e2def12d5bc1504232001de7441debed1b1cea8fb3b59b80c4a"}
Sep 18 18:01:12.150310 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.150257    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-gmztj" podStartSLOduration=1.875058093 podStartE2EDuration="5.150240398s" totalImagesPullingTime="3.275182305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:12.149475701 +0000 UTC m=+42.301350537" watchObservedRunningTime="2026-09-18 18:01:12.150240398 +0000 UTC m=+42.302115232"
Sep 18 18:01:12.256292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.256215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:12.405372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.405343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:12.502097 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.502063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:12.685638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.685610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:12.757622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:12.757593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:13.487472 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:13.487435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:13.675514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:13.675487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:13.675693 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:13.675549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:13.675693 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:13.675632    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:01:13.675693 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:13.675644    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:13.675693 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:13.675688    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls podName:542cb77a-84d0-433d-9d8f-36cb17a660f2 nodeName:}" failed. No retries permitted until 2026-09-18 18:01:21.675675155 +0000 UTC m=+51.827549969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls") pod "dns-default-jnwcp" (UID: "542cb77a-84d0-433d-9d8f-36cb17a660f2") : secret "dns-default-metrics-tls" not found
Sep 18 18:01:13.675904 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:01:13.675708    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls podName:a57d4563-a4fe-4512-92de-b33ab932fc1b nodeName:}" failed. No retries permitted until 2026-09-18 18:01:21.675694322 +0000 UTC m=+51.827569135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtwbq" (UID: "a57d4563-a4fe-4512-92de-b33ab932fc1b") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:01:13.883460 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:13.883368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:14.287378 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:14.287346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:14.484439 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:14.484410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:15.090886 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:15.090852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:15.487055 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:15.487002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:15.682184 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:15.682152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:15.737307 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:15.737234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:15.994267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:15.994156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:16.283735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:16.283652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:16.680698 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:16.680657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:16.737971 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:16.737930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:16.887354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:16.887267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:17.107713 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:17.107678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:17.172478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:17.172445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:17.701341 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:17.701308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:17.883390 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:17.883359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:18.201207 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:18.201179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:18.315277 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:18.315246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:18.682415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:18.682379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:19.090325 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:19.090251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:19.348806 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:19.348710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:19.483247 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:19.483213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:19.890847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:19.890805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:20.291982 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:20.291945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:20.482196 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:20.482157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:21.082770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.082732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:21.680662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.680628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:21.731770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.731738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:21.731906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.731799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:21.735155 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.735129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/542cb77a-84d0-433d-9d8f-36cb17a660f2-metrics-tls\") pod \"dns-default-jnwcp\" (UID: \"542cb77a-84d0-433d-9d8f-36cb17a660f2\") " pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:21.735251 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.735152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a57d4563-a4fe-4512-92de-b33ab932fc1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtwbq\" (UID: \"a57d4563-a4fe-4512-92de-b33ab932fc1b\") " pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:21.797735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.797705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:21.834260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.834237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtwbq"
Sep 18 18:01:21.881863 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:21.881814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:22.086493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.086410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:22.115356 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.115328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnwcp"]
Sep 18 18:01:22.118005 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:22.117976    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542cb77a_84d0_433d_9d8f_36cb17a660f2.slice/crio-caf29f43ed17c96090afea62772b2451f96a20da00fe37b0a4b8ccd8af2b2cf9 WatchSource:0}: Error finding container caf29f43ed17c96090afea62772b2451f96a20da00fe37b0a4b8ccd8af2b2cf9: Status 404 returned error can't find the container with id caf29f43ed17c96090afea62772b2451f96a20da00fe37b0a4b8ccd8af2b2cf9
Sep 18 18:01:22.139536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.139508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wtwbq"]
Sep 18 18:01:22.142733 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:22.142705    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57d4563_a4fe_4512_92de_b33ab932fc1b.slice/crio-c6aab548c9d7c2ecef4b4fe3a1519a46ed5e54d4bb0ffb529049070ab1253a0d WatchSource:0}: Error finding container c6aab548c9d7c2ecef4b4fe3a1519a46ed5e54d4bb0ffb529049070ab1253a0d: Status 404 returned error can't find the container with id c6aab548c9d7c2ecef4b4fe3a1519a46ed5e54d4bb0ffb529049070ab1253a0d
Sep 18 18:01:22.162430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.162404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtwbq" event={"ID":"a57d4563-a4fe-4512-92de-b33ab932fc1b","Type":"ContainerStarted","Data":"c6aab548c9d7c2ecef4b4fe3a1519a46ed5e54d4bb0ffb529049070ab1253a0d"}
Sep 18 18:01:22.163534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.163509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnwcp" event={"ID":"542cb77a-84d0-433d-9d8f-36cb17a660f2","Type":"ContainerStarted","Data":"caf29f43ed17c96090afea62772b2451f96a20da00fe37b0a4b8ccd8af2b2cf9"}
Sep 18 18:01:22.487961 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.487922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:22.687975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:22.687943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:23.168566 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:23.168525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtwbq" event={"ID":"a57d4563-a4fe-4512-92de-b33ab932fc1b","Type":"ContainerStarted","Data":"b97c07e6258950b05bbcec7dc267246bb030ebef2754229f9830bfdd8558702f"}
Sep 18 18:01:23.168736 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:23.168626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtwbq" event={"ID":"a57d4563-a4fe-4512-92de-b33ab932fc1b","Type":"ContainerStarted","Data":"c3ef559f72277692f4cb005f08a4375bfd02a716d671b3375d0821d2a1296671"}
Sep 18 18:01:24.174718 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:24.174683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnwcp" event={"ID":"542cb77a-84d0-433d-9d8f-36cb17a660f2","Type":"ContainerStarted","Data":"ac4d6ae576f8d64ec5d4fc4bb0e97bec822a22f681435548f9be0ca71899eb01"}
Sep 18 18:01:24.175133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:24.174747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnwcp" event={"ID":"542cb77a-84d0-433d-9d8f-36cb17a660f2","Type":"ContainerStarted","Data":"b90cbc2af82cccf99a47d860e6693006af6b58edda5e560df85e72d38d88e65a"}
Sep 18 18:01:24.175133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:24.174796    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:24.193829 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:24.193784    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jnwcp" podStartSLOduration=17.640118515 podStartE2EDuration="19.193769205s" totalImagesPullingTime="1.55365069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:24.192289631 +0000 UTC m=+54.344164466" watchObservedRunningTime="2026-09-18 18:01:24.193769205 +0000 UTC m=+54.345644040"
Sep 18 18:01:25.181866 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:25.181815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtwbq" event={"ID":"a57d4563-a4fe-4512-92de-b33ab932fc1b","Type":"ContainerStarted","Data":"3aaebb86f2a40e423d26d8f31430fa709a51b9d7305298ae7b5f060c21c9fd7e"}
Sep 18 18:01:25.202446 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:25.202374    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wtwbq" podStartSLOduration=17.990728351 podStartE2EDuration="20.202354937s" totalImagesPullingTime="2.211626586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:01:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:25.202075544 +0000 UTC m=+55.353950379" watchObservedRunningTime="2026-09-18 18:01:25.202354937 +0000 UTC m=+55.354229831"
Sep 18 18:01:27.030893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.030854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:27.129883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.129830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:27.183616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.183581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:27.328856 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.328768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:27.630082 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.629982    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"]
Sep 18 18:01:27.633171 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.633145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"]
Sep 18 18:01:27.633317 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.633297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.635970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.635951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dsvs2\""
Sep 18 18:01:27.636253 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.636237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.636892 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.636871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 18:01:27.636989 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.636945    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 18:01:27.637095 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.637008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 18:01:27.637151 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.637101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 18:01:27.638744 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.638677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 18:01:27.638850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.638761    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 18:01:27.638850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.638839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 18:01:27.639095 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.639078    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 18:01:27.658948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.658915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"]
Sep 18 18:01:27.659715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.659699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"]
Sep 18 18:01:27.674696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f6c855b66-7bvlm\" (UID: \"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.674845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.674845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-ca\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.674845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2md\" (UniqueName: \"kubernetes.io/projected/959dd40b-bab7-4d66-9fcb-403f150b59e5-kube-api-access-rp2md\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.675011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/959dd40b-bab7-4d66-9fcb-403f150b59e5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.675011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.675011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.674946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n8ht\" (UniqueName: \"kubernetes.io/projected/ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7-kube-api-access-4n8ht\") pod \"managed-serviceaccount-addon-agent-7f6c855b66-7bvlm\" (UID: \"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.675137 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.675013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-hub\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-hub\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f6c855b66-7bvlm\" (UID: \"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.775714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-ca\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2md\" (UniqueName: \"kubernetes.io/projected/959dd40b-bab7-4d66-9fcb-403f150b59e5-kube-api-access-rp2md\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/959dd40b-bab7-4d66-9fcb-403f150b59e5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.775714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.775640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8ht\" (UniqueName: \"kubernetes.io/projected/ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7-kube-api-access-4n8ht\") pod \"managed-serviceaccount-addon-agent-7f6c855b66-7bvlm\" (UID: \"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.776677 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.776645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/959dd40b-bab7-4d66-9fcb-403f150b59e5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.778293 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.778259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-ca\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.778413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.778321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-hub\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.778413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.778403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.778899 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.778876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/959dd40b-bab7-4d66-9fcb-403f150b59e5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.778960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.778892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f6c855b66-7bvlm\" (UID: \"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.796014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.795989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:27.798314 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.798294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2md\" (UniqueName: \"kubernetes.io/projected/959dd40b-bab7-4d66-9fcb-403f150b59e5-kube-api-access-rp2md\") pod \"cluster-proxy-proxy-agent-6b5986b54-h9lvr\" (UID: \"959dd40b-bab7-4d66-9fcb-403f150b59e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:27.799562 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.799542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8ht\" (UniqueName: \"kubernetes.io/projected/ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7-kube-api-access-4n8ht\") pod \"managed-serviceaccount-addon-agent-7f6c855b66-7bvlm\" (UID: \"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.913352 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.913309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:27.962663 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.962637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"
Sep 18 18:01:27.968216 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:27.968198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"
Sep 18 18:01:28.020173 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.019807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:28.130568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.130535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:28.237468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.237395    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm"]
Sep 18 18:01:28.240295 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:28.240261    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea3053c1_e0c6_4d2a_aac3_50d0ab4006d7.slice/crio-59a017b5c67fbec4ebce46d1a1e4db5b7d726faba76671d4758244eb73864bb4 WatchSource:0}: Error finding container 59a017b5c67fbec4ebce46d1a1e4db5b7d726faba76671d4758244eb73864bb4: Status 404 returned error can't find the container with id 59a017b5c67fbec4ebce46d1a1e4db5b7d726faba76671d4758244eb73864bb4
Sep 18 18:01:28.242114 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.241961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:28.249680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.249643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr"]
Sep 18 18:01:28.253002 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:28.252978    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod959dd40b_bab7_4d66_9fcb_403f150b59e5.slice/crio-d6fd7076035b8b47f5d96d282ca7fe0b444d0c83b70001c9d8d482fdbd4be789 WatchSource:0}: Error finding container d6fd7076035b8b47f5d96d282ca7fe0b444d0c83b70001c9d8d482fdbd4be789: Status 404 returned error can't find the container with id d6fd7076035b8b47f5d96d282ca7fe0b444d0c83b70001c9d8d482fdbd4be789
Sep 18 18:01:28.432956 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.432921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:28.532747 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.532666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:28.709350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.709314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:28.907478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:28.907366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:29.026092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:29.026054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:29.196774 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:29.196665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm" event={"ID":"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7","Type":"ContainerStarted","Data":"59a017b5c67fbec4ebce46d1a1e4db5b7d726faba76671d4758244eb73864bb4"}
Sep 18 18:01:29.198358 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:29.198307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr" event={"ID":"959dd40b-bab7-4d66-9fcb-403f150b59e5","Type":"ContainerStarted","Data":"d6fd7076035b8b47f5d96d282ca7fe0b444d0c83b70001c9d8d482fdbd4be789"}
Sep 18 18:01:30.143922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.143879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:30.245270 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.245234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:30.346776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.346728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:30.468128 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.468096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:30.553565 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.553526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:30.624431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.624398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:30.819628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:30.819588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:31.100584 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:31.100509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wppwj"
Sep 18 18:01:31.205479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:31.205445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm" event={"ID":"ea3053c1-e0c6-4d2a-aac3-50d0ab4006d7","Type":"ContainerStarted","Data":"0c92f8d09db260e5164e3aac0e5b2dfe1f68d7ee279af4d171deb44a27a62778"}
Sep 18 18:01:31.391305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:31.391213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:31.486434 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:31.486388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:31.994920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:31.994878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:32.096752 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:32.096715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:32.197311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:32.197278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:33.204197 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:33.204160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:33.603710 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:33.603620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:33.665669 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:33.665625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:34.184512 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:34.184481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jnwcp"
Sep 18 18:01:34.201197 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:34.201142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f6c855b66-7bvlm" podStartSLOduration=4.762424606 podStartE2EDuration="7.201123511s" totalImagesPullingTime="2.438698905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:31.239922916 +0000 UTC m=+61.391797751" watchObservedRunningTime="2026-09-18 18:01:34.201123511 +0000 UTC m=+64.352998347"
Sep 18 18:01:35.221513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.221476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr" event={"ID":"959dd40b-bab7-4d66-9fcb-403f150b59e5","Type":"ContainerStarted","Data":"5d20d512d375217f9f56cff85faa87164478f732ef4e355948c132ba47e05e11"}
Sep 18 18:01:35.221513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.221515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr" event={"ID":"959dd40b-bab7-4d66-9fcb-403f150b59e5","Type":"ContainerStarted","Data":"34b3cdadafe8cb724ccbb82c06fa65fb6fa524a8e59b3022dc07bddeffb8cf3c"}
Sep 18 18:01:35.222105 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.221530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr" event={"ID":"959dd40b-bab7-4d66-9fcb-403f150b59e5","Type":"ContainerStarted","Data":"a2c69295fc93671578c993ee045850c3e0d62d9dbb92b3dfa86c91ab24e7770c"}
Sep 18 18:01:35.239696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.239645    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5986b54-h9lvr" podStartSLOduration=1.9006146689999999 podStartE2EDuration="8.23962966s" totalImagesPullingTime="6.339014991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:35.238373658 +0000 UTC m=+65.390248493" watchObservedRunningTime="2026-09-18 18:01:35.23962966 +0000 UTC m=+65.391504495"
Sep 18 18:01:35.568739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.568652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:35.648280 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.648228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:35.903483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:35.903398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:36.643623 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.643580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:36.644108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.643644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:36.645824 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.645801    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 18:01:36.645937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.645892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 18:01:36.656722 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.656695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0daa103-d922-4223-bf67-567a3fb989c5-original-pull-secret\") pod \"global-pull-secret-syncer-qh4kd\" (UID: \"d0daa103-d922-4223-bf67-567a3fb989c5\") " pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:36.656834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.656814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce9057c-cad7-4604-9e93-b63faa1362ba-metrics-certs\") pod \"network-metrics-daemon-bs8m5\" (UID: \"4ce9057c-cad7-4604-9e93-b63faa1362ba\") " pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:36.744294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.744265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:36.746250 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.746231    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 18:01:36.756550 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.756525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 18:01:36.767415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.767389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb89t\" (UniqueName: \"kubernetes.io/projected/a79643fb-2296-47a6-9f3a-0246a0f76039-kube-api-access-wb89t\") pod \"network-check-target-lv9rm\" (UID: \"a79643fb-2296-47a6-9f3a-0246a0f76039\") " pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:36.824507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.824486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fqhj6\""
Sep 18 18:01:36.828951 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.828937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qh4kd"
Sep 18 18:01:36.833070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.833055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:36.914663 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.914595    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxcq\""
Sep 18 18:01:36.922506 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:36.922479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs8m5"
Sep 18 18:01:37.123000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.122971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qh4kd"]
Sep 18 18:01:37.126179 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:37.126151    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0daa103_d922_4223_bf67_567a3fb989c5.slice/crio-ee8dc158b5d35181dbc484619f9dc73684b2de2ea8fc593bd7c6bd4744c8dd12 WatchSource:0}: Error finding container ee8dc158b5d35181dbc484619f9dc73684b2de2ea8fc593bd7c6bd4744c8dd12: Status 404 returned error can't find the container with id ee8dc158b5d35181dbc484619f9dc73684b2de2ea8fc593bd7c6bd4744c8dd12
Sep 18 18:01:37.171388 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.171363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lv9rm"]
Sep 18 18:01:37.174249 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:37.174226    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79643fb_2296_47a6_9f3a_0246a0f76039.slice/crio-c189199dc8ebabc2c79adca422760487a07442999fec03bc24e118b408248254 WatchSource:0}: Error finding container c189199dc8ebabc2c79adca422760487a07442999fec03bc24e118b408248254: Status 404 returned error can't find the container with id c189199dc8ebabc2c79adca422760487a07442999fec03bc24e118b408248254
Sep 18 18:01:37.210295 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.210269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bs8m5"]
Sep 18 18:01:37.213212 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:37.213185    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce9057c_cad7_4604_9e93_b63faa1362ba.slice/crio-800b233ad0454db47f7c6d1dbb30cb726d9ba24add0fc12fc6e06a5a59f76471 WatchSource:0}: Error finding container 800b233ad0454db47f7c6d1dbb30cb726d9ba24add0fc12fc6e06a5a59f76471: Status 404 returned error can't find the container with id 800b233ad0454db47f7c6d1dbb30cb726d9ba24add0fc12fc6e06a5a59f76471
Sep 18 18:01:37.231791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.231757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lv9rm" event={"ID":"a79643fb-2296-47a6-9f3a-0246a0f76039","Type":"ContainerStarted","Data":"c189199dc8ebabc2c79adca422760487a07442999fec03bc24e118b408248254"}
Sep 18 18:01:37.232847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.232823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bs8m5" event={"ID":"4ce9057c-cad7-4604-9e93-b63faa1362ba","Type":"ContainerStarted","Data":"800b233ad0454db47f7c6d1dbb30cb726d9ba24add0fc12fc6e06a5a59f76471"}
Sep 18 18:01:37.233931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.233904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qh4kd" event={"ID":"d0daa103-d922-4223-bf67-567a3fb989c5","Type":"ContainerStarted","Data":"ee8dc158b5d35181dbc484619f9dc73684b2de2ea8fc593bd7c6bd4744c8dd12"}
Sep 18 18:01:37.534814 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:37.534719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:38.632800 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:38.632760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:39.244242 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:39.244207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bs8m5" event={"ID":"4ce9057c-cad7-4604-9e93-b63faa1362ba","Type":"ContainerStarted","Data":"d80e9891de06936d5e6b5b96e21159bdfe894785ed93117ead5d77e5147be74d"}
Sep 18 18:01:39.244437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:39.244278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bs8m5" event={"ID":"4ce9057c-cad7-4604-9e93-b63faa1362ba","Type":"ContainerStarted","Data":"f7af82d6e34ff6d2708bd6e1b864a358070ec09664ed157b161263ef0b37f8eb"}
Sep 18 18:01:39.261673 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:39.261616    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bs8m5" podStartSLOduration=65.69421948 podStartE2EDuration="1m7.26159578s" totalImagesPullingTime="1.5673763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:39.260943488 +0000 UTC m=+69.412818324" watchObservedRunningTime="2026-09-18 18:01:39.26159578 +0000 UTC m=+69.413470613"
Sep 18 18:01:39.345850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:39.345810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:39.445431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:39.445396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:39.509125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:39.509016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:40.489861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:40.489825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:40.589687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:40.589652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:40.707320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:40.707284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:40.830602 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:40.830516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:40.943394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:40.943359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:41.037625 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.037594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:41.042456 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.042419    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7q9vs"]
Sep 18 18:01:41.045813 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.045783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.048343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 18:01:41.048343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9xvzz\""
Sep 18 18:01:41.048343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 18:01:41.048551 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 18:01:41.048837 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048819    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 18:01:41.048914 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048863    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 18:01:41.048988 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.048976    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 18:01:41.138515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.138434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:41.179232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-wtmp\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eca1198-c433-41fd-a99a-fcdb01a75b2f-metrics-client-ca\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-textfile\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-sys\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-root\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-tls\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.179618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.179437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdnp\" (UniqueName: \"kubernetes.io/projected/0eca1198-c433-41fd-a99a-fcdb01a75b2f-kube-api-access-ztdnp\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.232443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.232406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:41.280307 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eca1198-c433-41fd-a99a-fcdb01a75b2f-metrics-client-ca\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-textfile\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-sys\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-root\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-sys\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-tls\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-textfile\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-root\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdnp\" (UniqueName: \"kubernetes.io/projected/0eca1198-c433-41fd-a99a-fcdb01a75b2f-kube-api-access-ztdnp\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.280730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-wtmp\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.281007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-wtmp\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.281007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.280965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eca1198-c433-41fd-a99a-fcdb01a75b2f-metrics-client-ca\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.281392 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.281371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.283395 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.283373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-tls\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.283635 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.283618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eca1198-c433-41fd-a99a-fcdb01a75b2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.289813 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.289788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdnp\" (UniqueName: \"kubernetes.io/projected/0eca1198-c433-41fd-a99a-fcdb01a75b2f-kube-api-access-ztdnp\") pod \"node-exporter-7q9vs\" (UID: \"0eca1198-c433-41fd-a99a-fcdb01a75b2f\") " pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.351961 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.351923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:41.357631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.357608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7q9vs"
Sep 18 18:01:41.692898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.692850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:41.973791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:41.973707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:42.043098 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:42.043039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:42.266933 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:42.266885    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eca1198_c433_41fd_a99a_fcdb01a75b2f.slice/crio-5852e42d5e744675769ae6864d6c792c220c6f3fced27f9d378c4e7992f8ef5f WatchSource:0}: Error finding container 5852e42d5e744675769ae6864d6c792c220c6f3fced27f9d378c4e7992f8ef5f: Status 404 returned error can't find the container with id 5852e42d5e744675769ae6864d6c792c220c6f3fced27f9d378c4e7992f8ef5f
Sep 18 18:01:42.289480 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:42.289455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:43.259761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.259716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lv9rm" event={"ID":"a79643fb-2296-47a6-9f3a-0246a0f76039","Type":"ContainerStarted","Data":"ca8858c6c71c83c2eff06221828d992d5d4ab97e5b649eb5aaae1e4b83b708c6"}
Sep 18 18:01:43.260207 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.259776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:01:43.261221 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.261197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qh4kd" event={"ID":"d0daa103-d922-4223-bf67-567a3fb989c5","Type":"ContainerStarted","Data":"4e0762dafa0b238e154c5e64f2fbe65ea2474eea9f8a35a9830a9d685a45de16"}
Sep 18 18:01:43.262282 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.262262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7q9vs" event={"ID":"0eca1198-c433-41fd-a99a-fcdb01a75b2f","Type":"ContainerStarted","Data":"5852e42d5e744675769ae6864d6c792c220c6f3fced27f9d378c4e7992f8ef5f"}
Sep 18 18:01:43.275461 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.275417    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lv9rm" podStartSLOduration=66.180558542 podStartE2EDuration="1m11.275402319s" totalImagesPullingTime="5.094843777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:43.274408576 +0000 UTC m=+73.426283416" watchObservedRunningTime="2026-09-18 18:01:43.275402319 +0000 UTC m=+73.427277157"
Sep 18 18:01:43.289628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.289564    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qh4kd" podStartSLOduration=66.13518749 podStartE2EDuration="1m11.289548022s" totalImagesPullingTime="5.154360532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:43.289480457 +0000 UTC m=+73.441355292" watchObservedRunningTime="2026-09-18 18:01:43.289548022 +0000 UTC m=+73.441422859"
Sep 18 18:01:43.308557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.308521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:43.546355 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.546326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:43.647623 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.647591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:43.942670 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:43.942634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:44.006436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.006381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:44.064898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.064862    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f948dfbf4-h9769"]
Sep 18 18:01:44.068205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.068183    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.070544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 18:01:44.070681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 18:01:44.070681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070583    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 18:01:44.070681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7gdsh\""
Sep 18 18:01:44.070681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 18:01:44.070888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070783    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 18:01:44.070940 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.070910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8pc505pik0fss\""
Sep 18 18:01:44.080818 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.080788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f948dfbf4-h9769"]
Sep 18 18:01:44.088477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.088448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:44.207074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.206951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.206995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-grpc-tls\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.207017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.207094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzgb\" (UniqueName: \"kubernetes.io/projected/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-kube-api-access-nmzgb\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.207154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.207181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-metrics-client-ca\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.207199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-tls\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.207354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.207217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.267376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.267345    2580 generic.go:356] "Generic (PLEG): container finished" podID="0eca1198-c433-41fd-a99a-fcdb01a75b2f" containerID="32492fc6eb3f8163363938fa11aa127b1105696bfc043eecf47e3cfe9b300686" exitCode=0
Sep 18 18:01:44.267766 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.267439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7q9vs" event={"ID":"0eca1198-c433-41fd-a99a-fcdb01a75b2f","Type":"ContainerDied","Data":"32492fc6eb3f8163363938fa11aa127b1105696bfc043eecf47e3cfe9b300686"}
Sep 18 18:01:44.290316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.290293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:44.307672 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzgb\" (UniqueName: \"kubernetes.io/projected/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-kube-api-access-nmzgb\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.307807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.307807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-metrics-client-ca\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.307807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-tls\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.307807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.308015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.308015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-grpc-tls\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.308015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.307902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.308567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.308540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-metrics-client-ca\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.310774 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.310747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.311001 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.310971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.311202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.311183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.311315 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.311292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-grpc-tls\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.311403 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.311293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.311495 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.311478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-secret-thanos-querier-tls\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.315371 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.315353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzgb\" (UniqueName: \"kubernetes.io/projected/300822b8-6ebc-41d7-a2e3-e9e96030d4d9-kube-api-access-nmzgb\") pod \"thanos-querier-6f948dfbf4-h9769\" (UID: \"300822b8-6ebc-41d7-a2e3-e9e96030d4d9\") " pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.377444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.377423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:44.662712 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:44.662683    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f948dfbf4-h9769"]
Sep 18 18:01:44.665450 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:44.665420    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300822b8_6ebc_41d7_a2e3_e9e96030d4d9.slice/crio-a5aba3be302d8cb3ce40e1eba27133aaa0028a0668c45e24294f187013955718 WatchSource:0}: Error finding container a5aba3be302d8cb3ce40e1eba27133aaa0028a0668c45e24294f187013955718: Status 404 returned error can't find the container with id a5aba3be302d8cb3ce40e1eba27133aaa0028a0668c45e24294f187013955718
Sep 18 18:01:45.275607 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.275572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7q9vs" event={"ID":"0eca1198-c433-41fd-a99a-fcdb01a75b2f","Type":"ContainerStarted","Data":"27592c80e98f428416f775b3acd68c11fbb4979badd7ca4177a37863c0d7f90c"}
Sep 18 18:01:45.276083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.275638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7q9vs" event={"ID":"0eca1198-c433-41fd-a99a-fcdb01a75b2f","Type":"ContainerStarted","Data":"085a3d4a0efbd916dd8a5cc7d92c2114e231110fc28119aab6c856185c96639c"}
Sep 18 18:01:45.276957 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.276928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"a5aba3be302d8cb3ce40e1eba27133aaa0028a0668c45e24294f187013955718"}
Sep 18 18:01:45.305638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.305596    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7q9vs" podStartSLOduration=2.962644931 podStartE2EDuration="4.30557936s" totalImagesPullingTime="1.274478206s" totalInitContainerRuntime="68.456223ms" isStatefulPod=true podCreationTimestamp="2026-09-18 18:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:45.303637019 +0000 UTC m=+75.455511855" watchObservedRunningTime="2026-09-18 18:01:45.30557936 +0000 UTC m=+75.457454194"
Sep 18 18:01:45.431606 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.431574    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"]
Sep 18 18:01:45.434566 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.434541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.437488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.437314    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 18:01:45.437488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.437333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 18:01:45.437488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.437429    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 18:01:45.437710 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.437507    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ec7dp73f3uuqi\""
Sep 18 18:01:45.437710 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.437515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 18:01:45.437859 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.437795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jnpk5\""
Sep 18 18:01:45.446424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.446403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"]
Sep 18 18:01:45.517750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7l8\" (UniqueName: \"kubernetes.io/projected/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-kube-api-access-xn7l8\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.517750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-secret-metrics-server-tls\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.517959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-audit-log\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.517959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-secret-metrics-server-client-certs\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.517959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-client-ca-bundle\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.517959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-metrics-server-audit-profiles\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.517959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.517916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.618757 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7l8\" (UniqueName: \"kubernetes.io/projected/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-kube-api-access-xn7l8\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.618757 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-secret-metrics-server-tls\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.618757 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-audit-log\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.619007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-secret-metrics-server-client-certs\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.619007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-client-ca-bundle\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.619007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-metrics-server-audit-profiles\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.619007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.618889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.619859 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.619828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.620014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.619948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-metrics-server-audit-profiles\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.620106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.620090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-audit-log\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.621772 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.621741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-secret-metrics-server-tls\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.622112 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.622096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-secret-metrics-server-client-certs\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.622234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.622215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-client-ca-bundle\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.627266 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.627245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7l8\" (UniqueName: \"kubernetes.io/projected/5b5385dc-804e-4017-b8c5-46e93f4c5d2f-kube-api-access-xn7l8\") pod \"metrics-server-78ccbd98dc-vnnh6\" (UID: \"5b5385dc-804e-4017-b8c5-46e93f4c5d2f\") " pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.745703 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.745669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:01:45.751970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.751869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:45.841962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:45.841920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:46.033924 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:46.033889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"]
Sep 18 18:01:46.036942 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:46.036910    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5385dc_804e_4017_b8c5_46e93f4c5d2f.slice/crio-a21b099f842660571bf653c79b98e9ad0bebebd94e9ba462b9a716f66a88125c WatchSource:0}: Error finding container a21b099f842660571bf653c79b98e9ad0bebebd94e9ba462b9a716f66a88125c: Status 404 returned error can't find the container with id a21b099f842660571bf653c79b98e9ad0bebebd94e9ba462b9a716f66a88125c
Sep 18 18:01:46.133294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:46.133254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:46.283182 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:46.283139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6" event={"ID":"5b5385dc-804e-4017-b8c5-46e93f4c5d2f","Type":"ContainerStarted","Data":"a21b099f842660571bf653c79b98e9ad0bebebd94e9ba462b9a716f66a88125c"}
Sep 18 18:01:46.613741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:46.613664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:46.737753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:46.737569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:47.130251 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.130214    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:47.205651 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.205612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:47.288337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.288302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"a830610f5448f23ff5e6f4efa5b47ede78644aae957ef3dc679f6eabe9ebce5c"}
Sep 18 18:01:47.288680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.288370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"3777cfb8b2b14f0a0e2fe128aaab9b9e3dd34e390d6a9371aaccb9a99e9f22c0"}
Sep 18 18:01:47.288680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.288399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"fd2d81d5c2e7e4d1cfa3cc494ce5b3118b24a93e4850df10fd3a88055cb75132"}
Sep 18 18:01:47.293278 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.293255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:01:47.295575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.295547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:47.296762 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.296745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.299922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.299891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 18:01:47.300056 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.300040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 18:01:47.300544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.300528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 18:01:47.300772 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.300757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7qzns\""
Sep 18 18:01:47.302304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302275    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 18:01:47.302418 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302334    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 18:01:47.302498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 18:01:47.302819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 18:01:47.302819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 18:01:47.302819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d4pq8j4hc7ji5\""
Sep 18 18:01:47.303222 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302819    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 18:01:47.303222 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.302946    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 18:01:47.303222 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.303159    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 18:01:47.303222 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.303175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 18:01:47.304700 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.304679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 18:01:47.314596 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.314579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 18:01:47.327452 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.327433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:01:47.431816 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.431943 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqm4m\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-kube-api-access-gqm4m\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.431943 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-config\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.431943 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.431943 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.431943 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.431955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-config-out\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-web-config\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.432691 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.432471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-config\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-config-out\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.533820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-web-config\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.533978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqm4m\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-kube-api-access-gqm4m\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.534920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.534905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.538215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.537432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.538215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.537772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-web-config\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.538215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.538154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.538600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.538561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.542877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.539174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.542877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.540517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.542877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.540986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.542877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.541391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.542877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.541690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.543200 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.543114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-config-out\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.543200 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.543176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.544743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.544316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.544743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.544479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.544743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.544697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-config\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.544969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.544942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.545140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.545117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.546087 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.546063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.546253 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.546225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqm4m\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-kube-api-access-gqm4m\") pod \"prometheus-k8s-0\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.605920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.605895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:01:47.894574 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:47.894493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:01:48.134341 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:01:48.134305    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce430c6_2b17_4b11_8be9_c961b52a1da6.slice/crio-8a29e141e27db2fd8b226b610152b7c3e9d1ad4f2bf8c98f650af4284c909fa9 WatchSource:0}: Error finding container 8a29e141e27db2fd8b226b610152b7c3e9d1ad4f2bf8c98f650af4284c909fa9: Status 404 returned error can't find the container with id 8a29e141e27db2fd8b226b610152b7c3e9d1ad4f2bf8c98f650af4284c909fa9
Sep 18 18:01:48.293493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:48.293452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"8a29e141e27db2fd8b226b610152b7c3e9d1ad4f2bf8c98f650af4284c909fa9"}
Sep 18 18:01:48.295061 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:48.295038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6" event={"ID":"5b5385dc-804e-4017-b8c5-46e93f4c5d2f","Type":"ContainerStarted","Data":"28e358ea74d85c242471f30b58de7c4511c43eb3a66169121c74c74775edb24b"}
Sep 18 18:01:48.311851 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:48.311804    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6" podStartSLOduration=1.172333942 podStartE2EDuration="3.311789922s" totalImagesPullingTime="2.13945598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:48.311143158 +0000 UTC m=+78.463017997" watchObservedRunningTime="2026-09-18 18:01:48.311789922 +0000 UTC m=+78.463664748"
Sep 18 18:01:49.301728 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:49.301692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"3876772f761e328b4d4b08235cfb65177bd2fb65a2e62f786fab8cd02994a88d"}
Sep 18 18:01:49.302115 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:49.301766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"ded2eebe0756773ff1274790fc3bf6231a4f7e762661933e744d2c5d28d10bf7"}
Sep 18 18:01:49.302115 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:49.301795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" event={"ID":"300822b8-6ebc-41d7-a2e3-e9e96030d4d9","Type":"ContainerStarted","Data":"33b2a53611a2c9eb452c771ebc2b24303f8b2a5baf4863350cdb0186a0fabc91"}
Sep 18 18:01:49.325863 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:49.325459    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769" podStartSLOduration=1.542432104 podStartE2EDuration="5.325437694s" totalImagesPullingTime="3.78300559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:49.323828003 +0000 UTC m=+79.475702838" watchObservedRunningTime="2026-09-18 18:01:49.325437694 +0000 UTC m=+79.477312528"
Sep 18 18:01:49.531956 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:49.531928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:50.108577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.108487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:50.168080 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.168015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:50.291127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.291081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:50.306933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.306907    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1" exitCode=0
Sep 18 18:01:50.307286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.306998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"}
Sep 18 18:01:50.307343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.307324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:50.479072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:50.479018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:51.318729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:51.318698    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f948dfbf4-h9769"
Sep 18 18:01:51.753559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:51.753526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:51.854833 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:51.854752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:52.353348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:52.353307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:52.453906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:52.453868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:53.416890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:53.416847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:53.517372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:53.517341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:54.325355 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.325269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"}
Sep 18 18:01:54.325355 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.325333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"}
Sep 18 18:01:54.325528 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.325374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"}
Sep 18 18:01:54.325528 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.325402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"}
Sep 18 18:01:54.325528 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.325425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"}
Sep 18 18:01:54.325528 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.325463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerStarted","Data":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"}
Sep 18 18:01:54.352403 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:54.352358    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.841680128 podStartE2EDuration="7.352343098s" totalImagesPullingTime="4.49235526s" totalInitContainerRuntime="18.30771ms" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:54.351748852 +0000 UTC m=+84.503623688" watchObservedRunningTime="2026-09-18 18:01:54.352343098 +0000 UTC m=+84.504217933"
Sep 18 18:01:55.419313 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:55.419273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:55.520849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:55.520807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:56.127289 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:56.127251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:56.218900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:56.218862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:57.606834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:01:57.606787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:02:02.201874 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:02.201821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:02.395862 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:02.395825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:02.522974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:02.522885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:02.613044 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:02.613005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:05.421366 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:05.421327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:05.519629 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:05.519599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:05.622846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:05.622804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:05.746667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:05.746600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:02:05.746667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:05.746677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:02:13.195581 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:13.195543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:14.270510 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:14.270480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lv9rm"
Sep 18 18:02:17.139397 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:17.139365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:17.254719 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:17.254678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:25.753091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:25.753058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:02:25.757683 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:25.757658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78ccbd98dc-vnnh6"
Sep 18 18:02:47.606927 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:47.606878    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:02:47.626838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:47.626807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:02:48.538499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:48.538472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:02:57.181302 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:57.181270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:57.343147 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:02:57.343119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:05.878561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.878520    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:03:05.879068 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.878935    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="prometheus" containerID="cri-o://53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b" gracePeriod=600
Sep 18 18:03:05.879068 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.878958    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="config-reloader" containerID="cri-o://979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6" gracePeriod=600
Sep 18 18:03:05.879068 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.878981    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy" containerID="cri-o://f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b" gracePeriod=600
Sep 18 18:03:05.879068 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.878942    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-web" containerID="cri-o://5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288" gracePeriod=600
Sep 18 18:03:05.879301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.878981    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068" gracePeriod=600
Sep 18 18:03:05.879301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:05.879178    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="thanos-sidecar" containerID="cri-o://d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830" gracePeriod=600
Sep 18 18:03:06.116821 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.116795    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.206138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-kubelet-serving-ca-bundle\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-metrics-client-certs\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-tls\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-2\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-serving-certs-ca-bundle\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-thanos-prometheus-http-client-file\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-trusted-ca-bundle\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-tls-assets\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-1\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206503    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-0\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-config\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqm4m\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-kube-api-access-gqm4m\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-db\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-grpc-tls\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-web-config\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-metrics-client-ca\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.206759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-config-out\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.207483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-kube-rbac-proxy\") pod \"dce430c6-2b17-4b11-8be9-c961b52a1da6\" (UID: \"dce430c6-2b17-4b11-8be9-c961b52a1da6\") "
Sep 18 18:03:06.207483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.206996    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.207483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.207011    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.207483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.207201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.208680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.208327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.208680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.208594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.209299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.209015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.209689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.209382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.209689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.209672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:03:06.210592 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.210566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:03:06.211259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.211259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-config" (OuterVolumeSpecName: "config") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.211259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.211259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:03:06.211905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.211905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-kube-api-access-gqm4m" (OuterVolumeSpecName: "kube-api-access-gqm4m") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "kube-api-access-gqm4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:03:06.212057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.211902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.212145 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.212122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.212410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.212393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.213336 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.213313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-config-out" (OuterVolumeSpecName: "config-out") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:03:06.224378 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.224356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-web-config" (OuterVolumeSpecName: "web-config") pod "dce430c6-2b17-4b11-8be9-c961b52a1da6" (UID: "dce430c6-2b17-4b11-8be9-c961b52a1da6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:06.307593 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307569    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307593 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307590    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307601    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-tls-assets\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307610    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307619    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307627    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307636    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-config\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307644    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307652    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqm4m\" (UniqueName: \"kubernetes.io/projected/dce430c6-2b17-4b11-8be9-c961b52a1da6-kube-api-access-gqm4m\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307660    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-db\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307668    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-grpc-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307675    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-web-config\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307682    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-configmap-metrics-client-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307689    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dce430c6-2b17-4b11-8be9-c961b52a1da6-config-out\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307697    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-kube-rbac-proxy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307706    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-metrics-client-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307715    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dce430c6-2b17-4b11-8be9-c961b52a1da6-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.307735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.307724    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dce430c6-2b17-4b11-8be9-c961b52a1da6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588366    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068" exitCode=0
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588394    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b" exitCode=0
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588406    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288" exitCode=0
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588415    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830" exitCode=0
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588426    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6" exitCode=0
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588433    2580 generic.go:356] "Generic (PLEG): container finished" podID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b" exitCode=0
Sep 18 18:03:06.588455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588450    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"}
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"}
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"}
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"}
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588677    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"}
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"}
Sep 18 18:03:06.588882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.588730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dce430c6-2b17-4b11-8be9-c961b52a1da6","Type":"ContainerDied","Data":"8a29e141e27db2fd8b226b610152b7c3e9d1ad4f2bf8c98f650af4284c909fa9"}
Sep 18 18:03:06.597235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.597215    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.604337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.604318    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.610703 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.610685    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.617312 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.617291    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.618497 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.618479    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:03:06.624828 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.624809    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.624918 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.624903    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:03:06.633429 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.633401    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.639907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.639892    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.640210 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.640187    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.640283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.640221    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"} err="failed to get container status \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist"
Sep 18 18:03:06.640283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.640264    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.640534 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.640514    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.640575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.640540    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"} err="failed to get container status \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist"
Sep 18 18:03:06.640575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.640556    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.640792 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.640773    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.640870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.640801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"} err="failed to get container status \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist"
Sep 18 18:03:06.640870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.640820    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.641091 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.641072    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.641138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641100    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"} err="failed to get container status \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist"
Sep 18 18:03:06.641138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641119    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.641367 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.641350    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.641410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641376    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"} err="failed to get container status \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist"
Sep 18 18:03:06.641410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641390    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.641601 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.641586    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.641664 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641607    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"} err="failed to get container status \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist"
Sep 18 18:03:06.641664 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641626    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.641853 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:03:06.641838    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.641894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641857    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"} err="failed to get container status \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist"
Sep 18 18:03:06.641894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.641870    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.642078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642062    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"} err="failed to get container status \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist"
Sep 18 18:03:06.642148 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642080    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.642326 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642309    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"} err="failed to get container status \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist"
Sep 18 18:03:06.642371 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642326    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.642554 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642536    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"} err="failed to get container status \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist"
Sep 18 18:03:06.642594 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642555    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.642799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642781    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"} err="failed to get container status \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist"
Sep 18 18:03:06.642836 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642801    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.643006 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.642990    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"} err="failed to get container status \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist"
Sep 18 18:03:06.643126 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643007    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.643287 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643270    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"} err="failed to get container status \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist"
Sep 18 18:03:06.643335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643287    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.643513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643489    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"} err="failed to get container status \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist"
Sep 18 18:03:06.643554 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643515    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.643736 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643719    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"} err="failed to get container status \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist"
Sep 18 18:03:06.643781 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643737    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.643957 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643935    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"} err="failed to get container status \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist"
Sep 18 18:03:06.643997 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.643959    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.644202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644186    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"} err="failed to get container status \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist"
Sep 18 18:03:06.644202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644202    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.644401 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644385    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"} err="failed to get container status \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist"
Sep 18 18:03:06.644439 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644402    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.644646 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644629    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"} err="failed to get container status \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist"
Sep 18 18:03:06.644689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644646    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.644878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644860    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"} err="failed to get container status \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist"
Sep 18 18:03:06.644931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.644878    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.645095 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645078    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"} err="failed to get container status \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist"
Sep 18 18:03:06.645147 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645096    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.645327 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645311    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"} err="failed to get container status \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist"
Sep 18 18:03:06.645375 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645328    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.645530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645514    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"} err="failed to get container status \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist"
Sep 18 18:03:06.645575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645530    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.645727 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645712    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"} err="failed to get container status \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist"
Sep 18 18:03:06.645773 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645728    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.645945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645925    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"} err="failed to get container status \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist"
Sep 18 18:03:06.645985 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.645946    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.646183 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646169    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"} err="failed to get container status \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist"
Sep 18 18:03:06.646237 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646183    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.646424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646406    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"} err="failed to get container status \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist"
Sep 18 18:03:06.646469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646426    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.646668 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646647    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"} err="failed to get container status \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist"
Sep 18 18:03:06.646729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646668    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.646878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646863    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"} err="failed to get container status \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist"
Sep 18 18:03:06.646878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.646877    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.647124 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647108    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"} err="failed to get container status \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist"
Sep 18 18:03:06.647178 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647124    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.647385 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647360    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"} err="failed to get container status \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist"
Sep 18 18:03:06.647385 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647382    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.647627 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647607    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"} err="failed to get container status \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist"
Sep 18 18:03:06.647698 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647628    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.647900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647874    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"} err="failed to get container status \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist"
Sep 18 18:03:06.647900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.647894    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.648178 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648154    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"} err="failed to get container status \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist"
Sep 18 18:03:06.648178 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648178    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.648448 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648427    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"} err="failed to get container status \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist"
Sep 18 18:03:06.648448 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648448    2580 scope.go:117] "RemoveContainer" containerID="905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"
Sep 18 18:03:06.648701 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068"} err="failed to get container status \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": rpc error: code = NotFound desc = could not find container \"905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068\": container with ID starting with 905497b31435d1030c1c39d11c73b5271457204a3aa84046feb3e082a2c72068 not found: ID does not exist"
Sep 18 18:03:06.648701 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648701    2580 scope.go:117] "RemoveContainer" containerID="f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"
Sep 18 18:03:06.648930 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648909    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b"} err="failed to get container status \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": rpc error: code = NotFound desc = could not find container \"f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b\": container with ID starting with f1fbfce20cbfbaec93b7e1e13850f79a71f7f487cea21b0e096da2e1c684908b not found: ID does not exist"
Sep 18 18:03:06.648973 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.648931    2580 scope.go:117] "RemoveContainer" containerID="5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"
Sep 18 18:03:06.649180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649160    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288"} err="failed to get container status \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": rpc error: code = NotFound desc = could not find container \"5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288\": container with ID starting with 5a0791ecc332d00747ee8379070e32a81eefa938e4c8d645e1c3ecf8ec0ad288 not found: ID does not exist"
Sep 18 18:03:06.649180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649180    2580 scope.go:117] "RemoveContainer" containerID="d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"
Sep 18 18:03:06.649435 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649417    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830"} err="failed to get container status \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": rpc error: code = NotFound desc = could not find container \"d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830\": container with ID starting with d2bdbd577ceb1ffb193c6a5b5dcc3f2d69c8ec5ab2d13564bb90c332e9a4f830 not found: ID does not exist"
Sep 18 18:03:06.649490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649436    2580 scope.go:117] "RemoveContainer" containerID="979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"
Sep 18 18:03:06.649652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649636    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6"} err="failed to get container status \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": rpc error: code = NotFound desc = could not find container \"979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6\": container with ID starting with 979f245601d5c1166dcbc78db51c2a8bacb9c62ba66f6f0de64f23543d06fcf6 not found: ID does not exist"
Sep 18 18:03:06.649694 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649655    2580 scope.go:117] "RemoveContainer" containerID="53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"
Sep 18 18:03:06.649885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649870    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b"} err="failed to get container status \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": rpc error: code = NotFound desc = could not find container \"53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b\": container with ID starting with 53d1d80c973e5cc90f1fcbbb1125c3caa485955dbbdb7d60b8e24d2f9efd807b not found: ID does not exist"
Sep 18 18:03:06.649885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.649885    2580 scope.go:117] "RemoveContainer" containerID="296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"
Sep 18 18:03:06.650129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.650113    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1"} err="failed to get container status \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": rpc error: code = NotFound desc = could not find container \"296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1\": container with ID starting with 296f044dace3dfa6a9fb942bf883536d6ab9073c223cf85ab0d0bdbca2ed2bb1 not found: ID does not exist"
Sep 18 18:03:06.660831 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.660814    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:03:06.661067 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661056    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="config-reloader"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661069    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="config-reloader"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661078    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="prometheus"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661083    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="prometheus"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661089    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-web"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661094    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-web"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661100    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="init-config-reloader"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661105    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="init-config-reloader"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661112    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="thanos-sidecar"
Sep 18 18:03:06.661120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661116    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="thanos-sidecar"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661147    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661152    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661158    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-thanos"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661163    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-thanos"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661206    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="prometheus"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661213    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-thanos"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661218    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="config-reloader"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661224    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy-web"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661231    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="kube-rbac-proxy"
Sep 18 18:03:06.661367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.661236    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" containerName="thanos-sidecar"
Sep 18 18:03:06.665980 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.665966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.668940 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.668917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 18:03:06.669019 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.669006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7qzns\""
Sep 18 18:03:06.669300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.669279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 18:03:06.669413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.669398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 18:03:06.669479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.669417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 18:03:06.669536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.669517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 18:03:06.670283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 18:03:06.670283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 18:03:06.670680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 18:03:06.670811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 18:03:06.670899 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670877    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d4pq8j4hc7ji5\""
Sep 18 18:03:06.670899 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 18:03:06.671056 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.670925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 18:03:06.671056 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.671000    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 18:03:06.673737 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.673719    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 18:03:06.675749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.675732    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 18:03:06.689044 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.689006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:03:06.710704 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.710807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.710807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-config-out\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.710807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-config\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.710807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.710807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-web-config\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.710990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xr4d\" (UniqueName: \"kubernetes.io/projected/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-kube-api-access-6xr4d\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.711305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.711193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812175 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812175 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812175 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812417 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812417 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-config-out\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812417 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-config\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-web-config\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xr4d\" (UniqueName: \"kubernetes.io/projected/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-kube-api-access-6xr4d\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.812797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.813161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.812883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.813220 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.813185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.814860 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.813510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.814860 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.814212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.814860 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.814474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.814860 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.814781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.815769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.815705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.815987 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.815884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.816401 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.816377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.816716 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.816695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.817428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.817408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.817527 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.817508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-config\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.817585 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.817544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.817797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.817769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-config-out\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.817976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.817960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.818011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.817970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.818075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.818047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.818799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.818783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-web-config\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.822638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.822620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xr4d\" (UniqueName: \"kubernetes.io/projected/cda83ab2-1ada-402e-9ee8-d8aa4a72a181-kube-api-access-6xr4d\") pod \"prometheus-k8s-0\" (UID: \"cda83ab2-1ada-402e-9ee8-d8aa4a72a181\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:06.976272 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:06.976243    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:07.277634 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:07.277607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 18:03:07.280084 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:03:07.280053    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda83ab2_1ada_402e_9ee8_d8aa4a72a181.slice/crio-eab35c83d0b3aa827a9cd2e50df866af12525c324c3f8739536adccfb60d75e7 WatchSource:0}: Error finding container eab35c83d0b3aa827a9cd2e50df866af12525c324c3f8739536adccfb60d75e7: Status 404 returned error can't find the container with id eab35c83d0b3aa827a9cd2e50df866af12525c324c3f8739536adccfb60d75e7
Sep 18 18:03:07.593751 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:07.593664    2580 generic.go:356] "Generic (PLEG): container finished" podID="cda83ab2-1ada-402e-9ee8-d8aa4a72a181" containerID="a962ba361b9396308a7ec697d12d5d35fade984f003f6574a75ac0da4cb48233" exitCode=0
Sep 18 18:03:07.593906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:07.593755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerDied","Data":"a962ba361b9396308a7ec697d12d5d35fade984f003f6574a75ac0da4cb48233"}
Sep 18 18:03:07.593906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:07.593830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"eab35c83d0b3aa827a9cd2e50df866af12525c324c3f8739536adccfb60d75e7"}
Sep 18 18:03:07.909390 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:07.909367    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce430c6-2b17-4b11-8be9-c961b52a1da6" path="/var/lib/kubelet/pods/dce430c6-2b17-4b11-8be9-c961b52a1da6/volumes"
Sep 18 18:03:08.601758 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.601719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"bcdadb2c0e4d8aa6df5f1ee6ac08c278baf05c811e94df5a48a736c09472e12a"}
Sep 18 18:03:08.602122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.601811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"5790f4e7d275fabd53af894fbdc6ed82559130f5f5794117e34d225b9303355e"}
Sep 18 18:03:08.602122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.601852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"376a4f17c7cd01441942ce472842580d565ef2dfcc570e60bcfe2fd1e472f937"}
Sep 18 18:03:08.602122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.601890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"1677e7a03daf01975b8f91d4f761e0a5c3c14671ba7a1a042c55795663db1b91"}
Sep 18 18:03:08.602122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.601931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"057c86a04d74957d5f03a5be215439ce734cc4b992a493af14cfae8366b630b0"}
Sep 18 18:03:08.602122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.601969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda83ab2-1ada-402e-9ee8-d8aa4a72a181","Type":"ContainerStarted","Data":"498bc5730e3d8f4cd4879dd06ea95c7a215a0792d89f42b112b68bb44475a1d5"}
Sep 18 18:03:08.637517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:08.637450    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.618385936 podStartE2EDuration="2.637418267s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.032331ms" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:08.634914499 +0000 UTC m=+158.786789334" watchObservedRunningTime="2026-09-18 18:03:08.637418267 +0000 UTC m=+158.789293103"
Sep 18 18:03:09.147443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:09.147417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:09.216178 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:09.216150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:11.977312 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:11.977284    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:03:17.120009 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:17.119916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:17.222531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:17.222497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:27.133362 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:27.133324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:27.209287 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:27.209254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:27.431127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:27.431096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:28.050186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:28.050151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:28.129048 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:28.128990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:37.794478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:37.794434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:37.895405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:37.895367    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:37.955453 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:37.955404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:39.897268 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:39.897229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:39.996018 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:39.995976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:45.172662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:45.172605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:45.235553 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:45.235523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:52.046363 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:03:52.046335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:06.977187 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:06.977149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:04:06.992679 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:06.992655    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:04:07.426358 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:07.426316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:07.822503 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:07.822433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 18:04:08.827218 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:08.827175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:11.019805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:11.019764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:11.131291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:11.131263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:14.561425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:14.561386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:14.661063 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:14.661013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.128236 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:15.128198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.290235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:15.290204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:16.568382 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:16.568344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:16.668276 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:16.668248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:16.946479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:16.946443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:19.491652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:19.491602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:23.825080 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:23.825018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:23.925215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:23.925186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:25.621372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:25.621331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:26.649820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:26.649777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:26.755551 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:26.755514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:29.795495 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:29.795459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:29.898042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:29.897992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:32.073526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:32.073498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:47.703530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:47.703450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:47.821198 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:47.821162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:48.374977 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:48.374931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:48.987837 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:48.987806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.093855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.093808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.174992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.174957    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"]
Sep 18 18:04:49.178340 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.178318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.182686 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.182657    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 18:04:49.182790 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.182753    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 18:04:49.182790 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.182751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 18:04:49.182790 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.182775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 18:04:49.182931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.182902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cj7q6\""
Sep 18 18:04:49.217045 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.215103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"]
Sep 18 18:04:49.242311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.242249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2f252eb-615a-453d-b179-9a17d7c25f27-manifests\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.242311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.242293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2f252eb-615a-453d-b179-9a17d7c25f27-tmp\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.242463 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.242341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ch6c\" (UniqueName: \"kubernetes.io/projected/c2f252eb-615a-453d-b179-9a17d7c25f27-kube-api-access-5ch6c\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.242463 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.242378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2f252eb-615a-453d-b179-9a17d7c25f27-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.242463 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.242407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2f252eb-615a-453d-b179-9a17d7c25f27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.342832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.342803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ch6c\" (UniqueName: \"kubernetes.io/projected/c2f252eb-615a-453d-b179-9a17d7c25f27-kube-api-access-5ch6c\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.342960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.342848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2f252eb-615a-453d-b179-9a17d7c25f27-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.342960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.342874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2f252eb-615a-453d-b179-9a17d7c25f27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.342960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.342924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2f252eb-615a-453d-b179-9a17d7c25f27-manifests\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.343184 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.342973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2f252eb-615a-453d-b179-9a17d7c25f27-tmp\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.343401 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.343373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2f252eb-615a-453d-b179-9a17d7c25f27-manifests\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.345263 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.345237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2f252eb-615a-453d-b179-9a17d7c25f27-tmp\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.345366 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.345277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2f252eb-615a-453d-b179-9a17d7c25f27-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.345366 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.345354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2f252eb-615a-453d-b179-9a17d7c25f27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.368601 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.368574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.373229 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.373211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ch6c\" (UniqueName: \"kubernetes.io/projected/c2f252eb-615a-453d-b179-9a17d7c25f27-kube-api-access-5ch6c\") pod \"opendatahub-operator-controller-manager-57cbb766c6-45zzb\" (UID: \"c2f252eb-615a-453d-b179-9a17d7c25f27\") " pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.487811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.487786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:49.599046 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.598994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.712976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.712939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.775367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.775292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"]
Sep 18 18:04:49.779980 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:04:49.779956    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f252eb_615a_453d_b179_9a17d7c25f27.slice/crio-44cda54f41b8068a62188a25fcf8ec82e34cb8639a3d72786fa06134de4d1337 WatchSource:0}: Error finding container 44cda54f41b8068a62188a25fcf8ec82e34cb8639a3d72786fa06134de4d1337: Status 404 returned error can't find the container with id 44cda54f41b8068a62188a25fcf8ec82e34cb8639a3d72786fa06134de4d1337
Sep 18 18:04:49.819345 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.819313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.924219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.924180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.960217 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:49.960193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb" event={"ID":"c2f252eb-615a-453d-b179-9a17d7c25f27","Type":"ContainerStarted","Data":"44cda54f41b8068a62188a25fcf8ec82e34cb8639a3d72786fa06134de4d1337"}
Sep 18 18:04:52.533764 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:52.533727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:52.693003 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:52.692180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:53.976334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:53.976300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb" event={"ID":"c2f252eb-615a-453d-b179-9a17d7c25f27","Type":"ContainerStarted","Data":"f60d1e79c62a07af3f452fbb5726ed4ddcce0256334d93a8e6c2a13c99a185ad"}
Sep 18 18:04:54.981698 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:54.981657    2580 generic.go:356] "Generic (PLEG): container finished" podID="c2f252eb-615a-453d-b179-9a17d7c25f27" containerID="f60d1e79c62a07af3f452fbb5726ed4ddcce0256334d93a8e6c2a13c99a185ad" exitCode=0
Sep 18 18:04:54.982201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:54.981743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb" event={"ID":"c2f252eb-615a-453d-b179-9a17d7c25f27","Type":"ContainerDied","Data":"f60d1e79c62a07af3f452fbb5726ed4ddcce0256334d93a8e6c2a13c99a185ad"}
Sep 18 18:04:55.986931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:55.986896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb" event={"ID":"c2f252eb-615a-453d-b179-9a17d7c25f27","Type":"ContainerStarted","Data":"ff0575807f408c736917a22dcd0f4f0d2ebffd8427f68b756aace1db6a51b191"}
Sep 18 18:04:55.987353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:55.987016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:04:56.018022 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:04:56.017976    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb" podStartSLOduration=2.413413128 podStartE2EDuration="7.017958062s" totalImagesPullingTime="3.57707323s" totalInitContainerRuntime="1.027471704s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:04:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:04:56.017638905 +0000 UTC m=+266.169513742" watchObservedRunningTime="2026-09-18 18:04:56.017958062 +0000 UTC m=+266.169832899"
Sep 18 18:05:06.992840 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:06.992813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57cbb766c6-45zzb"
Sep 18 18:05:09.095419 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:09.095382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:10.741567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:10.741526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:11.847317 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:11.847280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:12.022074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:12.022039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:13.307976 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:13.307940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:17.123238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:17.123199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:17.282766 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:17.282737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:18.113648 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:18.113620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:18.361907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:18.361865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:18.465360 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:18.465331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:19.939513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:19.939479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:20.004231 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:20.004199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:20.106215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:20.106177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:20.205930 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:20.205843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:20.306043 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:20.305995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:20.636695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:20.636609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:20.910750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:20.910716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:28.008826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.008790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:28.640815 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.640779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"]
Sep 18 18:05:28.644133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.644112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.646603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.646574    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 18:05:28.646746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.646622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 18:05:28.648712 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.648686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 18:05:28.648816 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.648717    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:05:28.648816 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.648758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 18:05:28.648816 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.648686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fq68h\""
Sep 18 18:05:28.662610 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.662584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"]
Sep 18 18:05:28.731204 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.731173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-845mr\" (UniqueName: \"kubernetes.io/projected/0c13bc09-19cd-4716-9519-653fdf1442ee-kube-api-access-845mr\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.731204 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.731211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c13bc09-19cd-4716-9519-653fdf1442ee-metrics-cert\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.731427 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.731252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c13bc09-19cd-4716-9519-653fdf1442ee-manager-config\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.731427 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.731341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c13bc09-19cd-4716-9519-653fdf1442ee-cert\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.832771 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.832732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-845mr\" (UniqueName: \"kubernetes.io/projected/0c13bc09-19cd-4716-9519-653fdf1442ee-kube-api-access-845mr\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.832771 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.832773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c13bc09-19cd-4716-9519-653fdf1442ee-metrics-cert\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.833050 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.832799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c13bc09-19cd-4716-9519-653fdf1442ee-manager-config\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.833050 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.832824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c13bc09-19cd-4716-9519-653fdf1442ee-cert\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.833563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.833542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c13bc09-19cd-4716-9519-653fdf1442ee-manager-config\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.835478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.835453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c13bc09-19cd-4716-9519-653fdf1442ee-metrics-cert\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.835555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.835512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c13bc09-19cd-4716-9519-653fdf1442ee-cert\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.849257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.849231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-845mr\" (UniqueName: \"kubernetes.io/projected/0c13bc09-19cd-4716-9519-653fdf1442ee-kube-api-access-845mr\") pod \"lws-controller-manager-6d5f876c6c-62fp8\" (UID: \"0c13bc09-19cd-4716-9519-653fdf1442ee\") " pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:28.953634 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:28.953613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:29.229873 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:29.229851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"]
Sep 18 18:05:29.232238 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:05:29.232214    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c13bc09_19cd_4716_9519_653fdf1442ee.slice/crio-a2f695f5907e82a9139f97ecf9c9861c14011d607e84b284c1d9f4478014bb4e WatchSource:0}: Error finding container a2f695f5907e82a9139f97ecf9c9861c14011d607e84b284c1d9f4478014bb4e: Status 404 returned error can't find the container with id a2f695f5907e82a9139f97ecf9c9861c14011d607e84b284c1d9f4478014bb4e
Sep 18 18:05:30.106707 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:30.106674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8" event={"ID":"0c13bc09-19cd-4716-9519-653fdf1442ee","Type":"ContainerStarted","Data":"a2f695f5907e82a9139f97ecf9c9861c14011d607e84b284c1d9f4478014bb4e"}
Sep 18 18:05:31.838283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:31.838228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:31.981563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:31.980658    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"]
Sep 18 18:05:31.990089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:31.988626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:31.992017 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:31.991932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 18 18:05:31.993707 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:31.993666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wvxrb\""
Sep 18 18:05:32.003654 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.003627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"]
Sep 18 18:05:32.018348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.018317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:32.027786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.027762    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"]
Sep 18 18:05:32.031436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.031417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.034550 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.034530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 18:05:32.035077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.034981    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pnrtm\""
Sep 18 18:05:32.055079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.055055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"]
Sep 18 18:05:32.061740 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.061714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5gj6\" (UniqueName: \"kubernetes.io/projected/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-kube-api-access-b5gj6\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.061865 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.061795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-cert\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.163145 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.163118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5gj6\" (UniqueName: \"kubernetes.io/projected/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-kube-api-access-b5gj6\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.163313 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.163172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sg84\" (UniqueName: \"kubernetes.io/projected/d9d560de-0a4a-4a13-ae2d-12a76fa58452-kube-api-access-2sg84\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.163313 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.163220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-cert\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.163420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.163379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d560de-0a4a-4a13-ae2d-12a76fa58452-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.163420 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:05:32.163384    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 18:05:32.163520 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:05:32.163444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-cert podName:f8842d59-6c7a-446a-a0ea-7cfa9702ec4e nodeName:}" failed. No retries permitted until 2026-09-18 18:05:32.663428038 +0000 UTC m=+302.815302851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-cert") pod "kserve-controller-manager-d6f4b65b8-cmr2b" (UID: "f8842d59-6c7a-446a-a0ea-7cfa9702ec4e") : secret "kserve-webhook-server-cert" not found
Sep 18 18:05:32.192021 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.191989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5gj6\" (UniqueName: \"kubernetes.io/projected/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-kube-api-access-b5gj6\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.264554 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.264519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sg84\" (UniqueName: \"kubernetes.io/projected/d9d560de-0a4a-4a13-ae2d-12a76fa58452-kube-api-access-2sg84\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.264734 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.264618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d560de-0a4a-4a13-ae2d-12a76fa58452-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.264850 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:05:32.264776    2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 18:05:32.264850 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:05:32.264846    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d560de-0a4a-4a13-ae2d-12a76fa58452-cert podName:d9d560de-0a4a-4a13-ae2d-12a76fa58452 nodeName:}" failed. No retries permitted until 2026-09-18 18:05:32.76482746 +0000 UTC m=+302.916702278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d9d560de-0a4a-4a13-ae2d-12a76fa58452-cert") pod "llmisvc-controller-manager-777c6bd7d5-pdpc5" (UID: "d9d560de-0a4a-4a13-ae2d-12a76fa58452") : secret "llmisvc-webhook-server-cert" not found
Sep 18 18:05:32.292193 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.292165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sg84\" (UniqueName: \"kubernetes.io/projected/d9d560de-0a4a-4a13-ae2d-12a76fa58452-kube-api-access-2sg84\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.359665 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.359626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:32.517544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.517459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:32.553817 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.553783    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 18:05:32.620173 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.620138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:32.667967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.667938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-cert\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.670442 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.670417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8842d59-6c7a-446a-a0ea-7cfa9702ec4e-cert\") pod \"kserve-controller-manager-d6f4b65b8-cmr2b\" (UID: \"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.768690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.768625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d560de-0a4a-4a13-ae2d-12a76fa58452-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.771104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.771078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d560de-0a4a-4a13-ae2d-12a76fa58452-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pdpc5\" (UID: \"d9d560de-0a4a-4a13-ae2d-12a76fa58452\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:32.905113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.905088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:32.943563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:32.943530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:33.121436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:33.121364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8" event={"ID":"0c13bc09-19cd-4716-9519-653fdf1442ee","Type":"ContainerStarted","Data":"3d5ee95cc9e6d2c12c7d8984d12a6a097bb149701ede4c29269d678fd7cfb6ea"}
Sep 18 18:05:33.121566 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:33.121496    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:33.152359 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:33.152316    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8" podStartSLOduration=1.794091945 podStartE2EDuration="5.15230079s" totalImagesPullingTime="3.358208845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:33.150218209 +0000 UTC m=+303.302093045" watchObservedRunningTime="2026-09-18 18:05:33.15230079 +0000 UTC m=+303.304175625"
Sep 18 18:05:33.206978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:33.206957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"]
Sep 18 18:05:33.209015 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:05:33.208982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8842d59_6c7a_446a_a0ea_7cfa9702ec4e.slice/crio-906382227f706d08dd493bdc19b716449f3dec9cce342193c3aed32b845acb4d WatchSource:0}: Error finding container 906382227f706d08dd493bdc19b716449f3dec9cce342193c3aed32b845acb4d: Status 404 returned error can't find the container with id 906382227f706d08dd493bdc19b716449f3dec9cce342193c3aed32b845acb4d
Sep 18 18:05:33.210424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:33.210408    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:05:33.228106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:33.228089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"]
Sep 18 18:05:33.229744 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:05:33.229722    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9d560de_0a4a_4a13_ae2d_12a76fa58452.slice/crio-16d3d165aeca1ee0234e53bfb37a2fb9410ad1846ef0dca0671aaca9be880ccc WatchSource:0}: Error finding container 16d3d165aeca1ee0234e53bfb37a2fb9410ad1846ef0dca0671aaca9be880ccc: Status 404 returned error can't find the container with id 16d3d165aeca1ee0234e53bfb37a2fb9410ad1846ef0dca0671aaca9be880ccc
Sep 18 18:05:34.134185 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:34.134063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b" event={"ID":"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e","Type":"ContainerStarted","Data":"906382227f706d08dd493bdc19b716449f3dec9cce342193c3aed32b845acb4d"}
Sep 18 18:05:34.136187 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:34.135879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5" event={"ID":"d9d560de-0a4a-4a13-ae2d-12a76fa58452","Type":"ContainerStarted","Data":"16d3d165aeca1ee0234e53bfb37a2fb9410ad1846ef0dca0671aaca9be880ccc"}
Sep 18 18:05:37.150858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:37.150820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5" event={"ID":"d9d560de-0a4a-4a13-ae2d-12a76fa58452","Type":"ContainerStarted","Data":"277dd0685c4372a70727f93ab0edaae3e97999ae01e1248b4f0793453d93907a"}
Sep 18 18:05:37.151450 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:37.151097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:05:37.153754 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:37.153718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b" event={"ID":"f8842d59-6c7a-446a-a0ea-7cfa9702ec4e","Type":"ContainerStarted","Data":"013253189feed6d12e12c924155d4f572b1c605cb2e1764690396d205e63a99f"}
Sep 18 18:05:37.153898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:37.153771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:05:37.183737 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:37.183690    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5" podStartSLOduration=2.7805522419999997 podStartE2EDuration="6.183677966s" totalImagesPullingTime="3.403125724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:37.178144508 +0000 UTC m=+307.330019354" watchObservedRunningTime="2026-09-18 18:05:37.183677966 +0000 UTC m=+307.335552794"
Sep 18 18:05:37.214398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:37.214348    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b" podStartSLOduration=2.7906008289999997 podStartE2EDuration="6.214333414s" totalImagesPullingTime="3.423732585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:37.21400253 +0000 UTC m=+307.365877365" watchObservedRunningTime="2026-09-18 18:05:37.214333414 +0000 UTC m=+307.366208248"
Sep 18 18:05:44.138591 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:44.138561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d5f876c6c-62fp8"
Sep 18 18:05:54.924496 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:54.924458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:55.022306 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:55.022271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:56.689898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:05:56.689862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:08.159678 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:08.159640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pdpc5"
Sep 18 18:06:08.162592 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:08.162571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cmr2b"
Sep 18 18:06:20.907515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:20.907429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:21.006512 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:21.006465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:36.302291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:36.302251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:38.877770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:38.877727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:38.974603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:38.974567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:39.315422 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:39.315390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:39.415429 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:39.415402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:39.704939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:39.704903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:40.954945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:40.954909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:42.843385 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.843357    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"]
Sep 18 18:06:42.846442 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.846427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:42.850009 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.849985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j6zlm\""
Sep 18 18:06:42.850290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.850274    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 18:06:42.850382 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.850328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 18:06:42.902140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.902112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"]
Sep 18 18:06:42.910662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.910631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:42.994188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.994161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35eb5fef-9900-4878-8c51-55d952299edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:42.994325 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:42.994190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzjt\" (UniqueName: \"kubernetes.io/projected/35eb5fef-9900-4878-8c51-55d952299edf-kube-api-access-7nzjt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:43.095462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.095398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35eb5fef-9900-4878-8c51-55d952299edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:43.095462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.095426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzjt\" (UniqueName: \"kubernetes.io/projected/35eb5fef-9900-4878-8c51-55d952299edf-kube-api-access-7nzjt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:43.095812 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.095788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35eb5fef-9900-4878-8c51-55d952299edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:43.103076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.103049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:43.106952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.106935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzjt\" (UniqueName: \"kubernetes.io/projected/35eb5fef-9900-4878-8c51-55d952299edf-kube-api-access-7nzjt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:43.156639 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.156615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:43.297278 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.297243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:43.434880 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:43.434858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"]
Sep 18 18:06:43.437588 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:06:43.437563    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eb5fef_9900_4878_8c51_55d952299edf.slice/crio-b6b2d0a34b948a0b04559fa4d3a2dfd41bc60d14259f20aae30b07912de64df6 WatchSource:0}: Error finding container b6b2d0a34b948a0b04559fa4d3a2dfd41bc60d14259f20aae30b07912de64df6: Status 404 returned error can't find the container with id b6b2d0a34b948a0b04559fa4d3a2dfd41bc60d14259f20aae30b07912de64df6
Sep 18 18:06:44.399566 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:44.399535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:44.409981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:44.409951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24" event={"ID":"35eb5fef-9900-4878-8c51-55d952299edf","Type":"ContainerStarted","Data":"b6b2d0a34b948a0b04559fa4d3a2dfd41bc60d14259f20aae30b07912de64df6"}
Sep 18 18:06:44.521572 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:44.521532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:45.906668 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:45.906600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:46.143298 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:46.143256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:47.599096 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.599059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"]
Sep 18 18:06:47.603009 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.602981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:06:47.608787 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.608681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-22rfh\""
Sep 18 18:06:47.703796 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.703769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"]
Sep 18 18:06:47.734384 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.734357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rkq\" (UniqueName: \"kubernetes.io/projected/a5a45bc6-d449-45b2-99d4-97e58e65bdaa-kube-api-access-r2rkq\") pod \"limitador-operator-controller-manager-74495c69f8-g4wk9\" (UID: \"a5a45bc6-d449-45b2-99d4-97e58e65bdaa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:06:47.835563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.835537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rkq\" (UniqueName: \"kubernetes.io/projected/a5a45bc6-d449-45b2-99d4-97e58e65bdaa-kube-api-access-r2rkq\") pod \"limitador-operator-controller-manager-74495c69f8-g4wk9\" (UID: \"a5a45bc6-d449-45b2-99d4-97e58e65bdaa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:06:47.843735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.843707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rkq\" (UniqueName: \"kubernetes.io/projected/a5a45bc6-d449-45b2-99d4-97e58e65bdaa-kube-api-access-r2rkq\") pod \"limitador-operator-controller-manager-74495c69f8-g4wk9\" (UID: \"a5a45bc6-d449-45b2-99d4-97e58e65bdaa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:06:47.916914 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:47.916887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:06:48.192826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:48.192804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"]
Sep 18 18:06:48.195207 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:06:48.195179    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a45bc6_d449_45b2_99d4_97e58e65bdaa.slice/crio-1107bc1401b6a2616b3e3a27d1d68f7c00c9f3dc879a6f80b1f6f1f4908e316b WatchSource:0}: Error finding container 1107bc1401b6a2616b3e3a27d1d68f7c00c9f3dc879a6f80b1f6f1f4908e316b: Status 404 returned error can't find the container with id 1107bc1401b6a2616b3e3a27d1d68f7c00c9f3dc879a6f80b1f6f1f4908e316b
Sep 18 18:06:48.429462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:48.429430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9" event={"ID":"a5a45bc6-d449-45b2-99d4-97e58e65bdaa","Type":"ContainerStarted","Data":"1107bc1401b6a2616b3e3a27d1d68f7c00c9f3dc879a6f80b1f6f1f4908e316b"}
Sep 18 18:06:48.431251 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:48.431224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24" event={"ID":"35eb5fef-9900-4878-8c51-55d952299edf","Type":"ContainerStarted","Data":"ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea"}
Sep 18 18:06:48.431400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:48.431352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:06:48.459270 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:48.459142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24" podStartSLOduration=2.155281461 podStartE2EDuration="6.459125861s" totalImagesPullingTime="4.3038444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:06:48.457096496 +0000 UTC m=+378.608971327" watchObservedRunningTime="2026-09-18 18:06:48.459125861 +0000 UTC m=+378.611000697"
Sep 18 18:06:48.931567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:48.931525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:49.033700 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:49.033667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:50.173013 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.172932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:50.290039 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.289991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:50.407349 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.407317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:50.442843 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.442779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9" event={"ID":"a5a45bc6-d449-45b2-99d4-97e58e65bdaa","Type":"ContainerStarted","Data":"da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42"}
Sep 18 18:06:50.442964 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.442929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:06:50.464245 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.464201    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9" podStartSLOduration=1.74472138 podStartE2EDuration="3.464187685s" totalImagesPullingTime="1.719466305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:06:50.46178008 +0000 UTC m=+380.613654915" watchObservedRunningTime="2026-09-18 18:06:50.464187685 +0000 UTC m=+380.616062520"
Sep 18 18:06:50.692057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.692009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:50.848312 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:50.848225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.321431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:54.321397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.427964 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:54.427933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.525841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:54.525805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.626602 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:54.626531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.725348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:54.725327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.827716 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:54.827693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:59.438986 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:06:59.438957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:01.449562 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.449528    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:01.584065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.584011    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"]
Sep 18 18:07:01.584269 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.584235    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24" podUID="35eb5fef-9900-4878-8c51-55d952299edf" containerName="manager" containerID="cri-o://ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea" gracePeriod=2
Sep 18 18:07:01.605869 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.605843    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"]
Sep 18 18:07:01.619355 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.619325    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"35eb5fef-9900-4878-8c51-55d952299edf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-b4q24\": pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:01.635242 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.635216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"]
Sep 18 18:07:01.635563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.635527    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" containerName="manager" containerID="cri-o://da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42" gracePeriod=2
Sep 18 18:07:01.652485 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.652346    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a5a45bc6-d449-45b2-99d4-97e58e65bdaa\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-g4wk9\": pods \"limitador-operator-controller-manager-74495c69f8-g4wk9\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:01.653172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.653148    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"]
Sep 18 18:07:01.654909 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.654875    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:01.687574 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.687551    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"]
Sep 18 18:07:01.687933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.687918    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" containerName="manager"
Sep 18 18:07:01.688001 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.687936    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" containerName="manager"
Sep 18 18:07:01.688001 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.687946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35eb5fef-9900-4878-8c51-55d952299edf" containerName="manager"
Sep 18 18:07:01.688001 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.687951    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35eb5fef-9900-4878-8c51-55d952299edf" containerName="manager"
Sep 18 18:07:01.688172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.688014    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35eb5fef-9900-4878-8c51-55d952299edf" containerName="manager"
Sep 18 18:07:01.688172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.688042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" containerName="manager"
Sep 18 18:07:01.691267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.691248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:01.712131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.712072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"]
Sep 18 18:07:01.728138 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.728112    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g4wk9\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:01.731740 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.731713    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:01.843558 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.843529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtqd\" (UniqueName: \"kubernetes.io/projected/cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022-kube-api-access-jmtqd\") pod \"limitador-operator-controller-manager-74495c69f8-6qj4f\" (UID: \"cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:01.879208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.879181    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"]
Sep 18 18:07:01.883094 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.883074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:01.885274 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.885244    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:01.887985 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:01.887956    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g4wk9\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:01.890089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.890069    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:01.901748 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.901726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"]
Sep 18 18:07:01.908968 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.908952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:01.944365 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.944342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtqd\" (UniqueName: \"kubernetes.io/projected/cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022-kube-api-access-jmtqd\") pod \"limitador-operator-controller-manager-74495c69f8-6qj4f\" (UID: \"cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:01.959492 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:01.959474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtqd\" (UniqueName: \"kubernetes.io/projected/cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022-kube-api-access-jmtqd\") pod \"limitador-operator-controller-manager-74495c69f8-6qj4f\" (UID: \"cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:02.018876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.018809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:02.044828 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.044808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2rkq\" (UniqueName: \"kubernetes.io/projected/a5a45bc6-d449-45b2-99d4-97e58e65bdaa-kube-api-access-r2rkq\") pod \"a5a45bc6-d449-45b2-99d4-97e58e65bdaa\" (UID: \"a5a45bc6-d449-45b2-99d4-97e58e65bdaa\") "
Sep 18 18:07:02.044949 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.044845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35eb5fef-9900-4878-8c51-55d952299edf-extensions-socket-volume\") pod \"35eb5fef-9900-4878-8c51-55d952299edf\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") "
Sep 18 18:07:02.044949 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.044929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nzjt\" (UniqueName: \"kubernetes.io/projected/35eb5fef-9900-4878-8c51-55d952299edf-kube-api-access-7nzjt\") pod \"35eb5fef-9900-4878-8c51-55d952299edf\" (UID: \"35eb5fef-9900-4878-8c51-55d952299edf\") "
Sep 18 18:07:02.045102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.045081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85jc\" (UniqueName: \"kubernetes.io/projected/1426b077-07c1-43a9-a1df-9bac8fd86d4f-kube-api-access-f85jc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4s45h\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.045397 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.045372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1426b077-07c1-43a9-a1df-9bac8fd86d4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4s45h\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.045510 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.045465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eb5fef-9900-4878-8c51-55d952299edf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "35eb5fef-9900-4878-8c51-55d952299edf" (UID: "35eb5fef-9900-4878-8c51-55d952299edf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:07:02.045598 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.045557    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35eb5fef-9900-4878-8c51-55d952299edf-extensions-socket-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:07:02.047431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.047403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a45bc6-d449-45b2-99d4-97e58e65bdaa-kube-api-access-r2rkq" (OuterVolumeSpecName: "kube-api-access-r2rkq") pod "a5a45bc6-d449-45b2-99d4-97e58e65bdaa" (UID: "a5a45bc6-d449-45b2-99d4-97e58e65bdaa"). InnerVolumeSpecName "kube-api-access-r2rkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:07:02.047747 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.047730    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eb5fef-9900-4878-8c51-55d952299edf-kube-api-access-7nzjt" (OuterVolumeSpecName: "kube-api-access-7nzjt") pod "35eb5fef-9900-4878-8c51-55d952299edf" (UID: "35eb5fef-9900-4878-8c51-55d952299edf"). InnerVolumeSpecName "kube-api-access-7nzjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:07:02.146927 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.146897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85jc\" (UniqueName: \"kubernetes.io/projected/1426b077-07c1-43a9-a1df-9bac8fd86d4f-kube-api-access-f85jc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4s45h\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.147057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.146941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1426b077-07c1-43a9-a1df-9bac8fd86d4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4s45h\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.147057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.146986    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nzjt\" (UniqueName: \"kubernetes.io/projected/35eb5fef-9900-4878-8c51-55d952299edf-kube-api-access-7nzjt\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:07:02.147057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.146996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2rkq\" (UniqueName: \"kubernetes.io/projected/a5a45bc6-d449-45b2-99d4-97e58e65bdaa-kube-api-access-r2rkq\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:07:02.147297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.147280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1426b077-07c1-43a9-a1df-9bac8fd86d4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4s45h\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.159859 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.159833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85jc\" (UniqueName: \"kubernetes.io/projected/1426b077-07c1-43a9-a1df-9bac8fd86d4f-kube-api-access-f85jc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4s45h\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.203209 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.203180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:02.319849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.319818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"]
Sep 18 18:07:02.321109 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:07:02.321074    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1d2fb1_f18d_4d2b_9fc8_5ec8b3474022.slice/crio-e7c8f75a168745fa1d70b8cb34c47aa8fae156ed56a255161cd2ed6988ca1c22 WatchSource:0}: Error finding container e7c8f75a168745fa1d70b8cb34c47aa8fae156ed56a255161cd2ed6988ca1c22: Status 404 returned error can't find the container with id e7c8f75a168745fa1d70b8cb34c47aa8fae156ed56a255161cd2ed6988ca1c22
Sep 18 18:07:02.484983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.484960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"]
Sep 18 18:07:02.487428 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:07:02.487405    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1426b077_07c1_43a9_a1df_9bac8fd86d4f.slice/crio-f47aeefb24b95659cc100d5c0d7eac713fed2ea8407519ac1cfd5a86132794d6 WatchSource:0}: Error finding container f47aeefb24b95659cc100d5c0d7eac713fed2ea8407519ac1cfd5a86132794d6: Status 404 returned error can't find the container with id f47aeefb24b95659cc100d5c0d7eac713fed2ea8407519ac1cfd5a86132794d6
Sep 18 18:07:02.501866 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.501836    2580 generic.go:356] "Generic (PLEG): container finished" podID="35eb5fef-9900-4878-8c51-55d952299edf" containerID="ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea" exitCode=0
Sep 18 18:07:02.501934 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.501881    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:02.501934 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.501928    2580 scope.go:117] "RemoveContainer" containerID="ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea"
Sep 18 18:07:02.503925 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:02.503900    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:02.504062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.503938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f" event={"ID":"cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022","Type":"ContainerStarted","Data":"174a85fb59daf3fd1601eedbaa5438ce3568a8081ef0b03c20f45ebae3d1dd7a"}
Sep 18 18:07:02.504062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.503983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f" event={"ID":"cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022","Type":"ContainerStarted","Data":"e7c8f75a168745fa1d70b8cb34c47aa8fae156ed56a255161cd2ed6988ca1c22"}
Sep 18 18:07:02.504062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.504048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:02.505540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.505521    2580 generic.go:356] "Generic (PLEG): container finished" podID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" containerID="da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42" exitCode=0
Sep 18 18:07:02.505622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.505566    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:02.507090 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.507065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h" event={"ID":"1426b077-07c1-43a9-a1df-9bac8fd86d4f","Type":"ContainerStarted","Data":"f47aeefb24b95659cc100d5c0d7eac713fed2ea8407519ac1cfd5a86132794d6"}
Sep 18 18:07:02.512099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.512082    2580 scope.go:117] "RemoveContainer" containerID="ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea"
Sep 18 18:07:02.512388 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:02.512369    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea\": container with ID starting with ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea not found: ID does not exist" containerID="ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea"
Sep 18 18:07:02.512462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.512395    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea"} err="failed to get container status \"ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea\": rpc error: code = NotFound desc = could not find container \"ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea\": container with ID starting with ae6268f1a92a08a80623be53e8cd330c97161ab4a82d3b784b6d4c3349867cea not found: ID does not exist"
Sep 18 18:07:02.512462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.512411    2580 scope.go:117] "RemoveContainer" containerID="da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42"
Sep 18 18:07:02.522338 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.522319    2580 scope.go:117] "RemoveContainer" containerID="da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42"
Sep 18 18:07:02.522621 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:02.522600    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42\": container with ID starting with da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42 not found: ID does not exist" containerID="da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42"
Sep 18 18:07:02.522714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.522628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42"} err="failed to get container status \"da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42\": rpc error: code = NotFound desc = could not find container \"da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42\": container with ID starting with da49d1a4f4c0b2407fd61cd2f2deddc218a11cb71eaa421613fdd29326104b42 not found: ID does not exist"
Sep 18 18:07:02.536489 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:02.534800    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:02.536860 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:02.536819    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g4wk9\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:02.538659 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:02.538624    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:02.995947 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:02.995912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:03.112868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.112832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:03.513438 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.513407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h" event={"ID":"1426b077-07c1-43a9-a1df-9bac8fd86d4f","Type":"ContainerStarted","Data":"837435882b7880be327e0488c39addb97ebffacfc32d5489163e4ffc5ebab4d0"}
Sep 18 18:07:03.513849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.513468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:03.520781 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:03.520749    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-b4q24\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="35eb5fef-9900-4878-8c51-55d952299edf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b4q24"
Sep 18 18:07:03.522555 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:07:03.522518    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g4wk9\" is forbidden: User \"system:node:ip-10-0-136-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-64.ec2.internal' and this object" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g4wk9"
Sep 18 18:07:03.549309 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.549262    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h" podStartSLOduration=2.549248014 podStartE2EDuration="2.549248014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:07:03.547377909 +0000 UTC m=+393.699252744" watchObservedRunningTime="2026-09-18 18:07:03.549248014 +0000 UTC m=+393.701122849"
Sep 18 18:07:03.549494 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.549475    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f" podStartSLOduration=2.549471217 podStartE2EDuration="2.549471217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:07:02.532833224 +0000 UTC m=+392.684708060" watchObservedRunningTime="2026-09-18 18:07:03.549471217 +0000 UTC m=+393.701346053"
Sep 18 18:07:03.557709 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.557680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:03.653721 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.653692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:03.909869 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.909839    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35eb5fef-9900-4878-8c51-55d952299edf" path="/var/lib/kubelet/pods/35eb5fef-9900-4878-8c51-55d952299edf/volumes"
Sep 18 18:07:03.910180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:03.910167    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a45bc6-d449-45b2-99d4-97e58e65bdaa" path="/var/lib/kubelet/pods/a5a45bc6-d449-45b2-99d4-97e58e65bdaa/volumes"
Sep 18 18:07:06.028504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:06.028478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:06.177888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:06.177861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:07.326944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:07.326901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:07.425508 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:07.425480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:11.744364 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:11.744323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:11.850692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:11.850663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:13.518239 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:13.518208    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6qj4f"
Sep 18 18:07:14.523137 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:14.523098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:07:16.443736 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:16.443705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:16.551400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:16.551368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:16.644161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:16.644133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:17.538083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:17.538010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:17.654473 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:17.654435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:26.747246 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:26.747206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:26.841424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:26.841392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:27.040733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:27.040648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:27.140636 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:27.140608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:29.747138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:29.747110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:29.854346 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:29.854310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:31.032322 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:31.032251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:31.276040 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:31.276003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:33.301676 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:33.301625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:33.398799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:33.398767    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:33.503611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:33.503572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:34.195445 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.195412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:34.296648 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.296612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:34.385862 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.385835    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-n9kf6"]
Sep 18 18:07:34.389452 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.389434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:07:34.391579 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.391558    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-px4dt\""
Sep 18 18:07:34.396830 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.396810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n9kf6"]
Sep 18 18:07:34.484053 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.483968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64btz\" (UniqueName: \"kubernetes.io/projected/ded4fb6a-40ff-4266-9351-936aeedc4f04-kube-api-access-64btz\") pod \"authorino-59c974bd48-n9kf6\" (UID: \"ded4fb6a-40ff-4266-9351-936aeedc4f04\") " pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:07:34.584477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.584445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64btz\" (UniqueName: \"kubernetes.io/projected/ded4fb6a-40ff-4266-9351-936aeedc4f04-kube-api-access-64btz\") pod \"authorino-59c974bd48-n9kf6\" (UID: \"ded4fb6a-40ff-4266-9351-936aeedc4f04\") " pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:07:34.593012 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.592990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64btz\" (UniqueName: \"kubernetes.io/projected/ded4fb6a-40ff-4266-9351-936aeedc4f04-kube-api-access-64btz\") pod \"authorino-59c974bd48-n9kf6\" (UID: \"ded4fb6a-40ff-4266-9351-936aeedc4f04\") " pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:07:34.699056 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.699016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:07:34.969934 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:34.969906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n9kf6"]
Sep 18 18:07:34.972466 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:07:34.972439    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podded4fb6a_40ff_4266_9351_936aeedc4f04.slice/crio-3cf54dd591ad2a0d3da317282cb821757ff8c7ea365d88a10579e0c7ff6cc8a4 WatchSource:0}: Error finding container 3cf54dd591ad2a0d3da317282cb821757ff8c7ea365d88a10579e0c7ff6cc8a4: Status 404 returned error can't find the container with id 3cf54dd591ad2a0d3da317282cb821757ff8c7ea365d88a10579e0c7ff6cc8a4
Sep 18 18:07:35.661927 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:35.661895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n9kf6" event={"ID":"ded4fb6a-40ff-4266-9351-936aeedc4f04","Type":"ContainerStarted","Data":"3cf54dd591ad2a0d3da317282cb821757ff8c7ea365d88a10579e0c7ff6cc8a4"}
Sep 18 18:07:37.672105 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:37.672069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n9kf6" event={"ID":"ded4fb6a-40ff-4266-9351-936aeedc4f04","Type":"ContainerStarted","Data":"5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10"}
Sep 18 18:07:54.477838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:54.477750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:54.578171 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:54.578132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:54.941570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:07:54.941536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:09.780349 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:09.780265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:09.908893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:09.908858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:11.895570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:11.895516    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-n9kf6" podStartSLOduration=36.163684334 podStartE2EDuration="37.895500969s" totalImagesPullingTime="1.731816635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:07:37.686611784 +0000 UTC m=+427.838486618" watchObservedRunningTime="2026-09-18 18:08:11.895500969 +0000 UTC m=+462.047375803"
Sep 18 18:08:11.896240 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:11.896223    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n9kf6"]
Sep 18 18:08:11.896398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:11.896379    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-n9kf6" podUID="ded4fb6a-40ff-4266-9351-936aeedc4f04" containerName="authorino" containerID="cri-o://5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10" gracePeriod=30
Sep 18 18:08:12.139968 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.139942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:08:12.274747 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.274721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64btz\" (UniqueName: \"kubernetes.io/projected/ded4fb6a-40ff-4266-9351-936aeedc4f04-kube-api-access-64btz\") pod \"ded4fb6a-40ff-4266-9351-936aeedc4f04\" (UID: \"ded4fb6a-40ff-4266-9351-936aeedc4f04\") "
Sep 18 18:08:12.276886 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.276863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded4fb6a-40ff-4266-9351-936aeedc4f04-kube-api-access-64btz" (OuterVolumeSpecName: "kube-api-access-64btz") pod "ded4fb6a-40ff-4266-9351-936aeedc4f04" (UID: "ded4fb6a-40ff-4266-9351-936aeedc4f04"). InnerVolumeSpecName "kube-api-access-64btz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:08:12.376016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.375991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64btz\" (UniqueName: \"kubernetes.io/projected/ded4fb6a-40ff-4266-9351-936aeedc4f04-kube-api-access-64btz\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:08:12.819142 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.819104    2580 generic.go:356] "Generic (PLEG): container finished" podID="ded4fb6a-40ff-4266-9351-936aeedc4f04" containerID="5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10" exitCode=0
Sep 18 18:08:12.819142 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.819141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n9kf6"
Sep 18 18:08:12.819335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.819153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n9kf6" event={"ID":"ded4fb6a-40ff-4266-9351-936aeedc4f04","Type":"ContainerDied","Data":"5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10"}
Sep 18 18:08:12.819335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.819186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n9kf6" event={"ID":"ded4fb6a-40ff-4266-9351-936aeedc4f04","Type":"ContainerDied","Data":"3cf54dd591ad2a0d3da317282cb821757ff8c7ea365d88a10579e0c7ff6cc8a4"}
Sep 18 18:08:12.819335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.819207    2580 scope.go:117] "RemoveContainer" containerID="5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10"
Sep 18 18:08:12.827846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.827829    2580 scope.go:117] "RemoveContainer" containerID="5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10"
Sep 18 18:08:12.828175 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:08:12.828156    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10\": container with ID starting with 5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10 not found: ID does not exist" containerID="5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10"
Sep 18 18:08:12.828243 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.828185    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10"} err="failed to get container status \"5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10\": rpc error: code = NotFound desc = could not find container \"5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10\": container with ID starting with 5ce638b80108c9f91f6435bc9e2289376191c594b75fc498590a5af0e43e5a10 not found: ID does not exist"
Sep 18 18:08:12.849677 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.849654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n9kf6"]
Sep 18 18:08:12.854900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:12.854882    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n9kf6"]
Sep 18 18:08:13.909393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:13.909358    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded4fb6a-40ff-4266-9351-936aeedc4f04" path="/var/lib/kubelet/pods/ded4fb6a-40ff-4266-9351-936aeedc4f04/volumes"
Sep 18 18:08:15.346392 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:15.346362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:16.914117 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:16.914081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:17.008705 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.008674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:17.114673 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.114640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:17.169252 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.169194    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-v82cs"]
Sep 18 18:08:17.169615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.169602    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded4fb6a-40ff-4266-9351-936aeedc4f04" containerName="authorino"
Sep 18 18:08:17.169656 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.169619    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded4fb6a-40ff-4266-9351-936aeedc4f04" containerName="authorino"
Sep 18 18:08:17.169720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.169709    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded4fb6a-40ff-4266-9351-936aeedc4f04" containerName="authorino"
Sep 18 18:08:17.174205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.174188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.179526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.179503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 18:08:17.180122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.180095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xxjjv\""
Sep 18 18:08:17.180229 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.180098    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 18:08:17.180720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.180644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 18:08:17.192797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.192776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-v82cs"]
Sep 18 18:08:17.223442 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.223415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:17.314149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.314117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.314257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.314152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sptn\" (UniqueName: \"kubernetes.io/projected/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-kube-api-access-2sptn\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.314257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.314183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-metrics-certs\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.415606 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.415578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.415759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.415611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sptn\" (UniqueName: \"kubernetes.io/projected/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-kube-api-access-2sptn\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.415759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.415641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-metrics-certs\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.415759 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:08:17.415721    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 18:08:17.415908 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:08:17.415785    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert podName:eb7bc87f-9018-4ae7-bb57-dc9b789a93a9 nodeName:}" failed. No retries permitted until 2026-09-18 18:08:17.915770325 +0000 UTC m=+468.067645142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert") pod "maas-controller-7dc7cc4dff-v82cs" (UID: "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9") : secret "maas-controller-webhook-cert" not found
Sep 18 18:08:17.418234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.418205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-metrics-certs\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.438141 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.438079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sptn\" (UniqueName: \"kubernetes.io/projected/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-kube-api-access-2sptn\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.638089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.638015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:17.919037 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.919010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:17.921278 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:17.921257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-v82cs\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") " pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:18.084230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:18.084197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:18.358919 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:18.358894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-v82cs"]
Sep 18 18:08:18.360606 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:08:18.360571    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7bc87f_9018_4ae7_bb57_dc9b789a93a9.slice/crio-c09bc17a10bf24528d4903604844e0741e3c2c59355f4083ada27d9ac957236f WatchSource:0}: Error finding container c09bc17a10bf24528d4903604844e0741e3c2c59355f4083ada27d9ac957236f: Status 404 returned error can't find the container with id c09bc17a10bf24528d4903604844e0741e3c2c59355f4083ada27d9ac957236f
Sep 18 18:08:18.850407 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:18.850368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs" event={"ID":"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9","Type":"ContainerStarted","Data":"c09bc17a10bf24528d4903604844e0741e3c2c59355f4083ada27d9ac957236f"}
Sep 18 18:08:20.672709 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:20.672682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:20.777796 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:20.777764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:20.861345 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:20.861260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs" event={"ID":"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9","Type":"ContainerStarted","Data":"555ab4ca927010b114ddf634959a33ff1fb328d61151dc3f159eb637e97c1e9a"}
Sep 18 18:08:20.861511 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:20.861384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:20.882472 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:20.882425    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs" podStartSLOduration=1.6835658850000002 podStartE2EDuration="3.882412147s" totalImagesPullingTime="2.198846262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:08:20.881687428 +0000 UTC m=+471.033562263" watchObservedRunningTime="2026-09-18 18:08:20.882412147 +0000 UTC m=+471.034286982"
Sep 18 18:08:31.867627 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:31.867600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:08:53.512998 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:53.512957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:53.632715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:53.632675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:55.020588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.020550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:55.161567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.161529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:55.280296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.280221    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"]
Sep 18 18:08:55.283562 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.283544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.286692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.286667    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:08:55.286876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.286863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 18:08:55.287513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.287498    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jkbfn\""
Sep 18 18:08:55.287565 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.287519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 18:08:55.287565 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.287523    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 18:08:55.303560 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.303535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"]
Sep 18 18:08:55.304394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.304366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:55.415254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.415219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:55.420297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.420272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgl2\" (UniqueName: \"kubernetes.io/projected/e873313f-0af1-4eec-9b30-e8019da9da23-kube-api-access-7rgl2\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.420430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.420314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-maas-api-tls\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.420430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.420379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873313f-0af1-4eec-9b30-e8019da9da23-openshift-service-ca\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.420430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.420396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-metrics-certs\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.521461 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.521428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873313f-0af1-4eec-9b30-e8019da9da23-openshift-service-ca\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.521608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.521467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-metrics-certs\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.521608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.521504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgl2\" (UniqueName: \"kubernetes.io/projected/e873313f-0af1-4eec-9b30-e8019da9da23-kube-api-access-7rgl2\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.521608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.521537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-maas-api-tls\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.522203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.522178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873313f-0af1-4eec-9b30-e8019da9da23-openshift-service-ca\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.524166 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.524139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-maas-api-tls\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.524245 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.524236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-metrics-certs\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.545010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.544960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgl2\" (UniqueName: \"kubernetes.io/projected/e873313f-0af1-4eec-9b30-e8019da9da23-kube-api-access-7rgl2\") pod \"maas-api-59f6566bbc-vndcr\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.593971 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.593936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:55.848097 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.845831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:55.877172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.877132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"]
Sep 18 18:08:55.880233 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:08:55.880205    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode873313f_0af1_4eec_9b30_e8019da9da23.slice/crio-482a4d981be5f6098358d2b73b9d8577b6955956fb525b47a3ec32bc1ef60fc5 WatchSource:0}: Error finding container 482a4d981be5f6098358d2b73b9d8577b6955956fb525b47a3ec32bc1ef60fc5: Status 404 returned error can't find the container with id 482a4d981be5f6098358d2b73b9d8577b6955956fb525b47a3ec32bc1ef60fc5
Sep 18 18:08:55.950161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:55.950127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:56.011061 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:56.011008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr" event={"ID":"e873313f-0af1-4eec-9b30-e8019da9da23","Type":"ContainerStarted","Data":"482a4d981be5f6098358d2b73b9d8577b6955956fb525b47a3ec32bc1ef60fc5"}
Sep 18 18:08:58.022233 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:58.022202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr" event={"ID":"e873313f-0af1-4eec-9b30-e8019da9da23","Type":"ContainerStarted","Data":"61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f"}
Sep 18 18:08:58.022613 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:58.022316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:08:58.043459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:08:58.043403    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr" podStartSLOduration=1.654858563 podStartE2EDuration="3.043384139s" totalImagesPullingTime="1.388525576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:08:58.041388248 +0000 UTC m=+508.193263101" watchObservedRunningTime="2026-09-18 18:08:58.043384139 +0000 UTC m=+508.195258975"
Sep 18 18:09:04.031299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:04.031271    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:09:05.069419 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:05.069362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:05.160332 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:05.160274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:08.740693 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:08.740653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:11.431775 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:11.431738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:14.062689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:14.062602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:14.680667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:14.680611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:25.802382 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:25.802327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:40.647681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.647652    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"]
Sep 18 18:09:40.651338 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.651323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.655188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.655163    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 18:09:40.655340 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.655222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v6bqs\""
Sep 18 18:09:40.655340 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.655301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 18:09:40.655474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.655460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 18:09:40.664754 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.664733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"]
Sep 18 18:09:40.699772 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.699746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a394e567-119d-44f4-83cc-686769c9b835-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.699905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.699788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.699905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.699807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.699981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.699911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.699981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.699941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.699981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.699970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xh8\" (UniqueName: \"kubernetes.io/projected/a394e567-119d-44f4-83cc-686769c9b835-kube-api-access-m5xh8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.706181 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.706144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:40.801243 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a394e567-119d-44f4-83cc-686769c9b835-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801407 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801407 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801527 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801527 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801527 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xh8\" (UniqueName: \"kubernetes.io/projected/a394e567-119d-44f4-83cc-686769c9b835-kube-api-access-m5xh8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.801926 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.801905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.803773 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.803751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a394e567-119d-44f4-83cc-686769c9b835-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.804052 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.804018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a394e567-119d-44f4-83cc-686769c9b835-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.809719 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.809696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xh8\" (UniqueName: \"kubernetes.io/projected/a394e567-119d-44f4-83cc-686769c9b835-kube-api-access-m5xh8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdrzt\" (UID: \"a394e567-119d-44f4-83cc-686769c9b835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:40.964099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:40.964063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:41.236483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:41.236439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"]
Sep 18 18:09:41.239089 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:09:41.239061    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394e567_119d_44f4_83cc_686769c9b835.slice/crio-c8f78e0029e130e3cdfd729f1a3add65971c70e03899b69cc64babf2b32a9fc2 WatchSource:0}: Error finding container c8f78e0029e130e3cdfd729f1a3add65971c70e03899b69cc64babf2b32a9fc2: Status 404 returned error can't find the container with id c8f78e0029e130e3cdfd729f1a3add65971c70e03899b69cc64babf2b32a9fc2
Sep 18 18:09:42.204285 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:42.204252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt" event={"ID":"a394e567-119d-44f4-83cc-686769c9b835","Type":"ContainerStarted","Data":"c8f78e0029e130e3cdfd729f1a3add65971c70e03899b69cc64babf2b32a9fc2"}
Sep 18 18:09:43.210551 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:43.210513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt" event={"ID":"a394e567-119d-44f4-83cc-686769c9b835","Type":"ContainerStarted","Data":"a1adc9a4f2e45e00b93113275a08d94b5f5606fd21d4e67fd2bd15ab9e2d1b7e"}
Sep 18 18:09:43.229367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:43.229317    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt" podStartSLOduration=2.196828312 podStartE2EDuration="3.229302536s" totalImagesPullingTime="1.032474224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:09:43.227397842 +0000 UTC m=+553.379272677" watchObservedRunningTime="2026-09-18 18:09:43.229302536 +0000 UTC m=+553.381177371"
Sep 18 18:09:50.964865 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:50.964826    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:50.965377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:50.964895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:50.977188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:50.977166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:51.256379 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:51.256297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdrzt"
Sep 18 18:09:55.802392 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:09:55.802355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:02.341370 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:02.341331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:14.033546 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.033516    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"]
Sep 18 18:10:14.037212 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.037194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.039855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.039841    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 18 18:10:14.048858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.048828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"]
Sep 18 18:10:14.102679 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.102644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:14.174399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.174364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.174399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.174401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e948230b-f428-4371-b97a-3cf09e5652a3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.174594 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.174441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.174594 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.174461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.174594 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.174477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7f6p\" (UniqueName: \"kubernetes.io/projected/e948230b-f428-4371-b97a-3cf09e5652a3-kube-api-access-r7f6p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.174594 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.174565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.275929 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.275898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.275938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.275964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e948230b-f428-4371-b97a-3cf09e5652a3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.276042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.276072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.276093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7f6p\" (UniqueName: \"kubernetes.io/projected/e948230b-f428-4371-b97a-3cf09e5652a3-kube-api-access-r7f6p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276397 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.276364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276452 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.276408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.276493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.276465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.278436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.278413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e948230b-f428-4371-b97a-3cf09e5652a3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.278544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.278530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e948230b-f428-4371-b97a-3cf09e5652a3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.284589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.284535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7f6p\" (UniqueName: \"kubernetes.io/projected/e948230b-f428-4371-b97a-3cf09e5652a3-kube-api-access-r7f6p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q24cv\" (UID: \"e948230b-f428-4371-b97a-3cf09e5652a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.347832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.347811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:14.629342 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:14.629263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"]
Sep 18 18:10:15.356197 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:15.356162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv" event={"ID":"e948230b-f428-4371-b97a-3cf09e5652a3","Type":"ContainerStarted","Data":"c025c832036f279b820b986d02da71516fd0f3bc733a44f09c6d5ccd9e2ef618"}
Sep 18 18:10:15.356568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:15.356212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv" event={"ID":"e948230b-f428-4371-b97a-3cf09e5652a3","Type":"ContainerStarted","Data":"4a4d328c5c237a139741557f3df7ed836671f759910bac3b7c760cb03c471729"}
Sep 18 18:10:15.397915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:15.397865    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv" podStartSLOduration=1.195281279 podStartE2EDuration="1.397850449s" totalImagesPullingTime="202.56917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:15.397176078 +0000 UTC m=+585.549050917" watchObservedRunningTime="2026-09-18 18:10:15.397850449 +0000 UTC m=+585.549725283"
Sep 18 18:10:24.348650 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:24.348613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:24.349150 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:24.348693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:24.361120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:24.361098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:24.405554 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:24.405526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q24cv"
Sep 18 18:10:38.699242 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:38.699201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:43.069533 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:43.069447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:50.309083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:10:50.309051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:43.276959 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:11:43.276888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:59.579293 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:12:59.579254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:59.736267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:12:59.736236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:59.803555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:12:59.803517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-v82cs"]
Sep 18 18:12:59.803767 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:12:59.803744    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs" podUID="eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" containerName="manager" containerID="cri-o://555ab4ca927010b114ddf634959a33ff1fb328d61151dc3f159eb637e97c1e9a" gracePeriod=30
Sep 18 18:12:59.846497 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:12:59.846424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:00.073067 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.073002    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" containerID="555ab4ca927010b114ddf634959a33ff1fb328d61151dc3f159eb637e97c1e9a" exitCode=0
Sep 18 18:13:00.073230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.073078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs" event={"ID":"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9","Type":"ContainerDied","Data":"555ab4ca927010b114ddf634959a33ff1fb328d61151dc3f159eb637e97c1e9a"}
Sep 18 18:13:00.153168 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.153146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:13:00.250991 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.250962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-metrics-certs\") pod \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") "
Sep 18 18:13:00.251161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.251114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert\") pod \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") "
Sep 18 18:13:00.251161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.251157    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sptn\" (UniqueName: \"kubernetes.io/projected/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-kube-api-access-2sptn\") pod \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\" (UID: \"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9\") "
Sep 18 18:13:00.253218 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.253187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" (UID: "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:13:00.253422 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.253399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" (UID: "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:13:00.253507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.253403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-kube-api-access-2sptn" (OuterVolumeSpecName: "kube-api-access-2sptn") pod "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" (UID: "eb7bc87f-9018-4ae7-bb57-dc9b789a93a9"). InnerVolumeSpecName "kube-api-access-2sptn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:13:00.352321 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.352283    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-webhook-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:13:00.352321 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.352312    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sptn\" (UniqueName: \"kubernetes.io/projected/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-kube-api-access-2sptn\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:13:00.352321 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:00.352321    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:13:01.079329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:01.079299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs" event={"ID":"eb7bc87f-9018-4ae7-bb57-dc9b789a93a9","Type":"ContainerDied","Data":"c09bc17a10bf24528d4903604844e0741e3c2c59355f4083ada27d9ac957236f"}
Sep 18 18:13:01.079329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:01.079324    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-v82cs"
Sep 18 18:13:01.079891 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:01.079362    2580 scope.go:117] "RemoveContainer" containerID="555ab4ca927010b114ddf634959a33ff1fb328d61151dc3f159eb637e97c1e9a"
Sep 18 18:13:01.102280 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:01.102252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-v82cs"]
Sep 18 18:13:01.114999 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:01.114977    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-v82cs"]
Sep 18 18:13:01.909628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:01.909595    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" path="/var/lib/kubelet/pods/eb7bc87f-9018-4ae7-bb57-dc9b789a93a9/volumes"
Sep 18 18:13:21.381083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:21.381051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:21.471870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:21.471837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:28.491130 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.491093    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"]
Sep 18 18:13:28.491515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.491441    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" containerName="manager"
Sep 18 18:13:28.491515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.491452    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" containerName="manager"
Sep 18 18:13:28.491604 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.491520    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7bc87f-9018-4ae7-bb57-dc9b789a93a9" containerName="manager"
Sep 18 18:13:28.494514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.494498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:28.496847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.496816    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 18 18:13:28.497048 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.497009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gtxjw\""
Sep 18 18:13:28.512961 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.512939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"]
Sep 18 18:13:28.553040 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.553009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcbq\" (UniqueName: \"kubernetes.io/projected/8c063afa-b3c7-4d78-8dd9-dd874ee7797f-kube-api-access-qxcbq\") pod \"ai-gateway-controller-7747fb5fbd-28sv2\" (UID: \"8c063afa-b3c7-4d78-8dd9-dd874ee7797f\") " pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:28.654335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.654308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcbq\" (UniqueName: \"kubernetes.io/projected/8c063afa-b3c7-4d78-8dd9-dd874ee7797f-kube-api-access-qxcbq\") pod \"ai-gateway-controller-7747fb5fbd-28sv2\" (UID: \"8c063afa-b3c7-4d78-8dd9-dd874ee7797f\") " pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:28.663931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.663911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcbq\" (UniqueName: \"kubernetes.io/projected/8c063afa-b3c7-4d78-8dd9-dd874ee7797f-kube-api-access-qxcbq\") pod \"ai-gateway-controller-7747fb5fbd-28sv2\" (UID: \"8c063afa-b3c7-4d78-8dd9-dd874ee7797f\") " pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:28.804546 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:28.804474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:29.091389 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:29.091362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"]
Sep 18 18:13:29.099223 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:29.099187    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:13:29.198377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:29.198349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2" event={"ID":"8c063afa-b3c7-4d78-8dd9-dd874ee7797f","Type":"ContainerStarted","Data":"a00e5f7d61642239690ceccf91bcece328be950e9ca3d4f5f386447435332d56"}
Sep 18 18:13:29.894130 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:29.894097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:31.211145 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:31.211105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2" event={"ID":"8c063afa-b3c7-4d78-8dd9-dd874ee7797f","Type":"ContainerStarted","Data":"43f51be913ec6c70781a5d828288121bd46443f3abaa670be1f6b2cd12bf25eb"}
Sep 18 18:13:31.211513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:31.211170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:31.228140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:31.228097    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2" podStartSLOduration=2.030280548 podStartE2EDuration="3.228082504s" totalImagesPullingTime="1.197801956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:13:31.227060571 +0000 UTC m=+781.378935406" watchObservedRunningTime="2026-09-18 18:13:31.228082504 +0000 UTC m=+781.379957342"
Sep 18 18:13:42.217755 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:42.217724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7747fb5fbd-28sv2"
Sep 18 18:13:43.529645 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:43.529595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:43.894589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:43.894510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:45.086717 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:45.086669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:45.185202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:45.185170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:46.196516 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:46.196475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:46.293674 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:13:46.293637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:01.600839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.600801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:01.696723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.696697    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:01.952134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.952101    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"]
Sep 18 18:14:01.956962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.956941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:01.960081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.960059    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 18:14:01.960197 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.960066    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 18:14:01.960264 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.960221    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 18:14:01.960457 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.960433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-039dbc-openshift-default-dockercfg-f7ntk\""
Sep 18 18:14:01.972928 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:01.972908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"]
Sep 18 18:14:02.022893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.022866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.131447 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istiod-ca-cert\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-credential-socket\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-data\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-podinfo\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-certs\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131856 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-token\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131856 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9ql\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-kube-api-access-qq9ql\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131856 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-socket\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.131856 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.131762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-envoy\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.132746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.132718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.214858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.214781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.232831 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.232803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-certs\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.232965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.232844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-token\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.232965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.232889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9ql\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-kube-api-access-qq9ql\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.232965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.232922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-socket\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.232971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-envoy\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istiod-ca-cert\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-credential-socket\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-data\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-podinfo\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-certs\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-credential-socket\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233806 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-data\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.233978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.233924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-socket\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.234221 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.234202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istiod-ca-cert\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.236039 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.236002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-envoy\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.236244 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.236227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-podinfo\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.242130 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.242108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9ql\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-kube-api-access-qq9ql\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.242456 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.242433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-token\") pod \"e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") " pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.267813 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.267786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:02.331073 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.331047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.632675 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.632608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"]
Sep 18 18:14:02.635511 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:14:02.635478    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ec0aca_3d3a_4a25_9d71_e80034f52d0a.slice/crio-93692ad7cc502721be8f4846ca0ba5a026eb3a61b657adef4908fe20011e2f3c WatchSource:0}: Error finding container 93692ad7cc502721be8f4846ca0ba5a026eb3a61b657adef4908fe20011e2f3c: Status 404 returned error can't find the container with id 93692ad7cc502721be8f4846ca0ba5a026eb3a61b657adef4908fe20011e2f3c
Sep 18 18:14:02.655550 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:02.655504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:03.163811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:03.163778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:03.259160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:03.259122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:03.352705 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:03.352670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" event={"ID":"00ec0aca-3d3a-4a25-9d71-e80034f52d0a","Type":"ContainerStarted","Data":"93692ad7cc502721be8f4846ca0ba5a026eb3a61b657adef4908fe20011e2f3c"}
Sep 18 18:14:03.637012 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:03.636877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:05.495213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:05.495173    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:14:05.495490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:05.495250    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:14:05.495490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:05.495277    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:14:05.546653 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:05.546608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:05.632466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:05.632373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:06.370576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.370537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" event={"ID":"00ec0aca-3d3a-4a25-9d71-e80034f52d0a","Type":"ContainerStarted","Data":"cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"}
Sep 18 18:14:06.397587 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.397540    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podStartSLOduration=2.540307065 podStartE2EDuration="5.397524704s" totalImagesPullingTime="2.857217639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:06.397242448 +0000 UTC m=+816.549117283" watchObservedRunningTime="2026-09-18 18:14:06.397524704 +0000 UTC m=+816.549399540"
Sep 18 18:14:06.787208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.787165    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"]
Sep 18 18:14:06.790937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.790912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:06.793307 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.793289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-37c153-serving-cert\""
Sep 18 18:14:06.802540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.802505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"]
Sep 18 18:14:06.831180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.831146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:06.926211 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.926177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:06.978952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.978926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:06.979093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.978956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6brp\" (UniqueName: \"kubernetes.io/projected/87b3bce8-e817-4fa2-8b60-584ae6a4508e-kube-api-access-h6brp\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:06.979093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.979067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b3bce8-e817-4fa2-8b60-584ae6a4508e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:06.979186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:06.979100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-metrics-certs\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.029810 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.029773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:07.080353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.080298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b3bce8-e817-4fa2-8b60-584ae6a4508e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.080353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.080331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-metrics-certs\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.080504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.080377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.080504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.080408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6brp\" (UniqueName: \"kubernetes.io/projected/87b3bce8-e817-4fa2-8b60-584ae6a4508e-kube-api-access-h6brp\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.081128 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.081002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b3bce8-e817-4fa2-8b60-584ae6a4508e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.083112 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.083089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-metrics-certs\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.083193 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.083089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.089904 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.089883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6brp\" (UniqueName: \"kubernetes.io/projected/87b3bce8-e817-4fa2-8b60-584ae6a4508e-kube-api-access-h6brp\") pod \"maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.103707 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.103688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:07.268122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.268083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:07.272790 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.272765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:07.375504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.375437    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:07.376320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.376301    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:07.405143 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.405122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"]
Sep 18 18:14:07.407581 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:14:07.407552    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b3bce8_e817_4fa2_8b60_584ae6a4508e.slice/crio-8445b240b81ab00c11d205222349fe6df7c91b59c2e86543eb567594314438c5 WatchSource:0}: Error finding container 8445b240b81ab00c11d205222349fe6df7c91b59c2e86543eb567594314438c5: Status 404 returned error can't find the container with id 8445b240b81ab00c11d205222349fe6df7c91b59c2e86543eb567594314438c5
Sep 18 18:14:07.498505 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.498471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:07.609423 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:07.609361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:08.138426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.138351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:08.382371 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.382332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k" event={"ID":"87b3bce8-e817-4fa2-8b60-584ae6a4508e","Type":"ContainerStarted","Data":"e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a"}
Sep 18 18:14:08.382579 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.382390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k" event={"ID":"87b3bce8-e817-4fa2-8b60-584ae6a4508e","Type":"ContainerStarted","Data":"8445b240b81ab00c11d205222349fe6df7c91b59c2e86543eb567594314438c5"}
Sep 18 18:14:08.382662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.382630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:08.424474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.424417    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k" podStartSLOduration=2.10179845 podStartE2EDuration="2.424398288s" totalImagesPullingTime="322.599838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:08.423602313 +0000 UTC m=+818.575477149" watchObservedRunningTime="2026-09-18 18:14:08.424398288 +0000 UTC m=+818.576273123"
Sep 18 18:14:08.609957 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.609918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:08.821200 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:08.821120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:09.066969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:09.066930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:09.339967 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:09.339921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.204021 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.203989    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"]
Sep 18 18:14:10.207522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.207504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.212690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.212663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-039dbc-serving-cert\""
Sep 18 18:14:10.230829 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.230713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"]
Sep 18 18:14:10.253264 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.253224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.313208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.313181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-metrics-certs\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.313333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.313226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a2da44-97d4-47fa-b6ee-e7763deb8617-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.313333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.313293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-maas-api-tls\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.313333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.313323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqq7\" (UniqueName: \"kubernetes.io/projected/84a2da44-97d4-47fa-b6ee-e7763deb8617-kube-api-access-zxqq7\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.353180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.353147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.414347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.414328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-metrics-certs\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.414468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.414366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a2da44-97d4-47fa-b6ee-e7763deb8617-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.414468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.414399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-maas-api-tls\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.414468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.414418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqq7\" (UniqueName: \"kubernetes.io/projected/84a2da44-97d4-47fa-b6ee-e7763deb8617-kube-api-access-zxqq7\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.415110 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.415088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a2da44-97d4-47fa-b6ee-e7763deb8617-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.416864 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.416833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-metrics-certs\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.417014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.416995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-maas-api-tls\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.423356 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.423335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqq7\" (UniqueName: \"kubernetes.io/projected/84a2da44-97d4-47fa-b6ee-e7763deb8617-kube-api-access-zxqq7\") pod \"maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.452538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.452512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.527099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.527013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:10.765363 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.765332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.810344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.810317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"]
Sep 18 18:14:10.812607 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:14:10.812580    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a2da44_97d4_47fa_b6ee_e7763deb8617.slice/crio-cd0b5386a512222acb334a7719ad3eeca825c3714a04fd10d029389e55ff9bfe WatchSource:0}: Error finding container cd0b5386a512222acb334a7719ad3eeca825c3714a04fd10d029389e55ff9bfe: Status 404 returned error can't find the container with id cd0b5386a512222acb334a7719ad3eeca825c3714a04fd10d029389e55ff9bfe
Sep 18 18:14:10.862410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:10.862373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.195507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.195476    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"]
Sep 18 18:14:11.200078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.200055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.202121 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.202097    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-de7535-serving-cert\""
Sep 18 18:14:11.210078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.209921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"]
Sep 18 18:14:11.220805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.220780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.220903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.220886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e256921-a139-4dd7-b89c-e4bcd45096a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.220969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.220918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-metrics-certs\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.221019 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.220976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6256\" (UniqueName: \"kubernetes.io/projected/0e256921-a139-4dd7-b89c-e4bcd45096a2-kube-api-access-q6256\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.251792 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.251761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.322069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.321458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e256921-a139-4dd7-b89c-e4bcd45096a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.322069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.321500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-metrics-certs\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.322069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.321541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6256\" (UniqueName: \"kubernetes.io/projected/0e256921-a139-4dd7-b89c-e4bcd45096a2-kube-api-access-q6256\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.322069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.321602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.322069 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:11.321730    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-serving-cert: secret "maas-api-e2e-inf-a-de7535-serving-cert" not found
Sep 18 18:14:11.322069 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:11.321796    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls podName:0e256921-a139-4dd7-b89c-e4bcd45096a2 nodeName:}" failed. No retries permitted until 2026-09-18 18:14:11.821781485 +0000 UTC m=+821.973656298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls") pod "maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" (UID: "0e256921-a139-4dd7-b89c-e4bcd45096a2") : secret "maas-api-e2e-inf-a-de7535-serving-cert" not found
Sep 18 18:14:11.322589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.322165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e256921-a139-4dd7-b89c-e4bcd45096a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.324413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.324389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-metrics-certs\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.332423 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.332399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6256\" (UniqueName: \"kubernetes.io/projected/0e256921-a139-4dd7-b89c-e4bcd45096a2-kube-api-access-q6256\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.344758 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.344730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.398933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.398899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q" event={"ID":"84a2da44-97d4-47fa-b6ee-e7763deb8617","Type":"ContainerStarted","Data":"172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567"}
Sep 18 18:14:11.399337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.398953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q" event={"ID":"84a2da44-97d4-47fa-b6ee-e7763deb8617","Type":"ContainerStarted","Data":"cd0b5386a512222acb334a7719ad3eeca825c3714a04fd10d029389e55ff9bfe"}
Sep 18 18:14:11.399337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.398999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:11.418569 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.418392    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q" podStartSLOduration=0.959277485 podStartE2EDuration="1.418375291s" totalImagesPullingTime="459.097806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:11.417377656 +0000 UTC m=+821.569252493" watchObservedRunningTime="2026-09-18 18:14:11.418375291 +0000 UTC m=+821.570250127"
Sep 18 18:14:11.738432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.738402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.827090 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.827058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.829881 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.829858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:11.888343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.888300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.982072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:11.982018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:12.113702 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:12.113626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:12.349820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:12.349779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:12.421799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:12.421771    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"]
Sep 18 18:14:12.423597 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:14:12.423569    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e256921_a139_4dd7_b89c_e4bcd45096a2.slice/crio-726070abb8a0f5e16fd19bad7ebbd8ea0418e4e69ea8f56f83dc6c0f5ebdae6a WatchSource:0}: Error finding container 726070abb8a0f5e16fd19bad7ebbd8ea0418e4e69ea8f56f83dc6c0f5ebdae6a: Status 404 returned error can't find the container with id 726070abb8a0f5e16fd19bad7ebbd8ea0418e4e69ea8f56f83dc6c0f5ebdae6a
Sep 18 18:14:12.468779 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:12.468744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:12.566763 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:12.566716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:12.680490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:12.680396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:13.183128 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.183088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:13.271263 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.271233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:13.410389 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.410354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" event={"ID":"0e256921-a139-4dd7-b89c-e4bcd45096a2","Type":"ContainerStarted","Data":"587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da"}
Sep 18 18:14:13.410590 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.410410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" event={"ID":"0e256921-a139-4dd7-b89c-e4bcd45096a2","Type":"ContainerStarted","Data":"726070abb8a0f5e16fd19bad7ebbd8ea0418e4e69ea8f56f83dc6c0f5ebdae6a"}
Sep 18 18:14:13.410590 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.410496    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:13.466179 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.466077    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" podStartSLOduration=2.064454345 podStartE2EDuration="2.466059653s" totalImagesPullingTime="401.605308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:13.465426344 +0000 UTC m=+823.617301182" watchObservedRunningTime="2026-09-18 18:14:13.466059653 +0000 UTC m=+823.617934488"
Sep 18 18:14:13.529937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:13.529863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:14.323603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:14.323563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:14.393351 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:14.393324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:14:15.314562 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:15.314523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:15.963316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:15.963268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:16.060511 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:16.060477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:16.530425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:16.530387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:17.409599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:17.409570    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:14:17.450006 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:17.449953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:18.455018 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:18.454973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:18.789706 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:18.789612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:18.885092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:18.885054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:19.080076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:19.079974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:19.421489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:19.421458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:14:19.659741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:19.659701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:20.236301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.236268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:20.480701 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.480665    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"]
Sep 18 18:14:20.484359 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.484336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.486421 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.486355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7fbf2b1b-serving-cert\""
Sep 18 18:14:20.494647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.494621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"]
Sep 18 18:14:20.515616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.515588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a607d-3127-43ae-b18c-9dc84f52857c-openshift-service-ca\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.515758 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.515725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prfz9\" (UniqueName: \"kubernetes.io/projected/3e9a607d-3127-43ae-b18c-9dc84f52857c-kube-api-access-prfz9\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.515821 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.515785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-metrics-certs\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.515885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.515819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.592332 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.592295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:20.616404 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.616372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfz9\" (UniqueName: \"kubernetes.io/projected/3e9a607d-3127-43ae-b18c-9dc84f52857c-kube-api-access-prfz9\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.616536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.616427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-metrics-certs\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.616600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.616563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.616659 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.616623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a607d-3127-43ae-b18c-9dc84f52857c-openshift-service-ca\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.616740 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:20.616721    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-serving-cert: secret "maas-api-e2e-ait-7fbf2b1b-serving-cert" not found
Sep 18 18:14:20.616818 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:20.616805    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls podName:3e9a607d-3127-43ae-b18c-9dc84f52857c nodeName:}" failed. No retries permitted until 2026-09-18 18:14:21.116786165 +0000 UTC m=+831.268660980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls") pod "maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" (UID: "3e9a607d-3127-43ae-b18c-9dc84f52857c") : secret "maas-api-e2e-ait-7fbf2b1b-serving-cert" not found
Sep 18 18:14:20.617359 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.617324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a607d-3127-43ae-b18c-9dc84f52857c-openshift-service-ca\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.619260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.619217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-metrics-certs\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.648734 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.648699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfz9\" (UniqueName: \"kubernetes.io/projected/3e9a607d-3127-43ae-b18c-9dc84f52857c-kube-api-access-prfz9\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:20.703343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.703253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:20.879615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:20.879522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:21.015047 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:21.014989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:21.120928 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:21.120896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:21.123599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:21.123573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls\") pod \"maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:21.395852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:21.395815    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:21.696176 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:21.694635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"]
Sep 18 18:14:21.717967 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:14:21.699470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9a607d_3127_43ae_b18c_9dc84f52857c.slice/crio-4e534f057a5e965d7976b3a9025915af4db11e0148e1ac39daf44222bb2d2d83 WatchSource:0}: Error finding container 4e534f057a5e965d7976b3a9025915af4db11e0148e1ac39daf44222bb2d2d83: Status 404 returned error can't find the container with id 4e534f057a5e965d7976b3a9025915af4db11e0148e1ac39daf44222bb2d2d83
Sep 18 18:14:21.944945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:21.944877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:22.051238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.051139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:22.169935 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.169907    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"]
Sep 18 18:14:22.173406 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.173387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.178310 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.178293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7fbf2b1b\""
Sep 18 18:14:22.178413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.178376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7fbf2b1b-dockercfg-4xnzc\""
Sep 18 18:14:22.191878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.191850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"]
Sep 18 18:14:22.234770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.234740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj2hg\" (UniqueName: \"kubernetes.io/projected/678ace46-6812-45c2-9a74-ba9baa7c4f60-kube-api-access-lj2hg\") pod \"payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.234770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.234773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678ace46-6812-45c2-9a74-ba9baa7c4f60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.336402 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.336309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2hg\" (UniqueName: \"kubernetes.io/projected/678ace46-6812-45c2-9a74-ba9baa7c4f60-kube-api-access-lj2hg\") pod \"payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.336402 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.336346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678ace46-6812-45c2-9a74-ba9baa7c4f60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.336968 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.336946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678ace46-6812-45c2-9a74-ba9baa7c4f60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.349437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.349417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj2hg\" (UniqueName: \"kubernetes.io/projected/678ace46-6812-45c2-9a74-ba9baa7c4f60-kube-api-access-lj2hg\") pod \"payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.376439 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.376409    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 18:14:22.376531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.376455    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 18:14:22.464371 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.464329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" event={"ID":"3e9a607d-3127-43ae-b18c-9dc84f52857c","Type":"ContainerStarted","Data":"cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0"}
Sep 18 18:14:22.464516 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.464383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" event={"ID":"3e9a607d-3127-43ae-b18c-9dc84f52857c","Type":"ContainerStarted","Data":"4e534f057a5e965d7976b3a9025915af4db11e0148e1ac39daf44222bb2d2d83"}
Sep 18 18:14:22.464516 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.464410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:22.466009 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.465984    2580 generic.go:356] "Generic (PLEG): container finished" podID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerID="cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631" exitCode=0
Sep 18 18:14:22.466119 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.466060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" event={"ID":"00ec0aca-3d3a-4a25-9d71-e80034f52d0a","Type":"ContainerDied","Data":"cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"}
Sep 18 18:14:22.466384 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.466370    2580 scope.go:117] "RemoveContainer" containerID="cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"
Sep 18 18:14:22.468462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.468435    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:14:22.468531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.468516    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:14:22.468575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.468546    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:14:22.482822 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.482803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:22.563108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.563044    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" podStartSLOduration=2.204489686 podStartE2EDuration="2.563003403s" totalImagesPullingTime="358.513717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:22.523755598 +0000 UTC m=+832.675630445" watchObservedRunningTime="2026-09-18 18:14:22.563003403 +0000 UTC m=+832.714878243"
Sep 18 18:14:22.611531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.611445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:22.766395 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:22.766359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"]
Sep 18 18:14:23.273100 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.273057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:23.375391 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.375308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:23.475498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.474725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" event={"ID":"00ec0aca-3d3a-4a25-9d71-e80034f52d0a","Type":"ContainerStarted","Data":"21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb"}
Sep 18 18:14:23.476254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.476193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc" event={"ID":"678ace46-6812-45c2-9a74-ba9baa7c4f60","Type":"ContainerStarted","Data":"7ecd29ad145a7bb8109475e83d9290f677e5d34ac5aec12c859a4b77ec9b9e14"}
Sep 18 18:14:23.671654 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.671614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:23.808386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.808297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:23.905339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:23.905280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:24.159530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:24.159439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:24.268844 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:24.268757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:24.269050 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:24.268936    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 18:14:24.269114 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:24.269007    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 18:14:24.669049 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:24.668957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:24.851949 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:24.851786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:25.268923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:25.268887    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 18:14:25.269127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:25.268954    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 18:14:25.853705 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:25.853657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:26.268662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.268485    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 18:14:26.268662 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.268567    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 18:14:26.414915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.414871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:26.494665 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.494626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc" event={"ID":"678ace46-6812-45c2-9a74-ba9baa7c4f60","Type":"ContainerStarted","Data":"2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4"}
Sep 18 18:14:26.494848 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.494832    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:26.540754 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.540672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:26.564585 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.564531    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc" podStartSLOduration=1.467640055 podStartE2EDuration="4.564513688s" totalImagesPullingTime="3.096873633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:26.563859819 +0000 UTC m=+836.715734655" watchObservedRunningTime="2026-09-18 18:14:26.564513688 +0000 UTC m=+836.716388524"
Sep 18 18:14:26.674708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.674668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:26.788647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:26.788614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:27.068775 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:27.068736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:27.268767 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:27.268733    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 18:14:27.268937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:27.268817    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 18:14:27.695775 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:27.695742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:27.794516 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:27.794478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:28.082866 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:28.082773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:28.268739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:28.268697    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 18:14:28.268928 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:28.268775    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 18:14:28.482224 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:28.482198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:29.225062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:29.225005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:29.272942 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:29.272917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:29.273364 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:29.273340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:29.274215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:29.274197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:14:29.848122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:29.848081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:30.519141 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:30.519107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:31.206187 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.206158    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"]
Sep 18 18:14:31.214394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.214373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.217230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.217204    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bcf02f-serving-cert\""
Sep 18 18:14:31.222732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.222312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"]
Sep 18 18:14:31.250666 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.250640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:31.328126 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.328082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.328254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.328168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd437dfb-6a7e-495a-bb02-95eedec06b28-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.328254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.328216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-metrics-certs\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.328254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.328241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwtdh\" (UniqueName: \"kubernetes.io/projected/dd437dfb-6a7e-495a-bb02-95eedec06b28-kube-api-access-wwtdh\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.349966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.349933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:31.429434 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.429409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.429556 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.429467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd437dfb-6a7e-495a-bb02-95eedec06b28-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.429556 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.429504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-metrics-certs\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.429556 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.429537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwtdh\" (UniqueName: \"kubernetes.io/projected/dd437dfb-6a7e-495a-bb02-95eedec06b28-kube-api-access-wwtdh\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.429760 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:31.429569    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-serving-cert: secret "maas-api-e2e-inf-b-bcf02f-serving-cert" not found
Sep 18 18:14:31.429760 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:31.429647    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls podName:dd437dfb-6a7e-495a-bb02-95eedec06b28 nodeName:}" failed. No retries permitted until 2026-09-18 18:14:31.92962637 +0000 UTC m=+842.081501202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls") pod "maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" (UID: "dd437dfb-6a7e-495a-bb02-95eedec06b28") : secret "maas-api-e2e-inf-b-bcf02f-serving-cert" not found
Sep 18 18:14:31.430381 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.430357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd437dfb-6a7e-495a-bb02-95eedec06b28-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.432611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.432592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-metrics-certs\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.438965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.438940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwtdh\" (UniqueName: \"kubernetes.io/projected/dd437dfb-6a7e-495a-bb02-95eedec06b28-kube-api-access-wwtdh\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.507381 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.507312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:31.717179 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.717140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:31.817400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.817327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:31.934209 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.934172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:31.936828 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:31.936797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:32.127765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:32.127681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:32.157438 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:32.157392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:32.403688 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:32.403662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"]
Sep 18 18:14:32.405934 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:14:32.405903    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd437dfb_6a7e_495a_bb02_95eedec06b28.slice/crio-f65eeae6894d94d274b4d43851355c6ff52ffa33364f10bd024466cd84d4d9b0 WatchSource:0}: Error finding container f65eeae6894d94d274b4d43851355c6ff52ffa33364f10bd024466cd84d4d9b0: Status 404 returned error can't find the container with id f65eeae6894d94d274b4d43851355c6ff52ffa33364f10bd024466cd84d4d9b0
Sep 18 18:14:32.523554 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:32.523513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" event={"ID":"dd437dfb-6a7e-495a-bb02-95eedec06b28","Type":"ContainerStarted","Data":"f65eeae6894d94d274b4d43851355c6ff52ffa33364f10bd024466cd84d4d9b0"}
Sep 18 18:14:33.035304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.035273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:33.492066 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.492021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:33.531595 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.531567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" event={"ID":"dd437dfb-6a7e-495a-bb02-95eedec06b28","Type":"ContainerStarted","Data":"1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57"}
Sep 18 18:14:33.531760 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.531687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:33.570741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.570699    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" podStartSLOduration=2.128201198 podStartE2EDuration="2.570685908s" totalImagesPullingTime="442.48471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:33.567989372 +0000 UTC m=+843.719864207" watchObservedRunningTime="2026-09-18 18:14:33.570685908 +0000 UTC m=+843.722560742"
Sep 18 18:14:33.590777 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.590744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:33.975599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:33.975567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:37.501894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:37.501860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:39.542600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:39.542570    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:14:42.527937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:42.527893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:44.097221 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:44.097172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:44.948933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:44.948898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:45.686486 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:45.686443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:46.615765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:46.615734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:47.411305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:47.411272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:47.511933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:47.511901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:48.215077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:48.215045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:48.950646 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:48.950605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:49.732899 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:49.732859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:50.418545 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:50.418505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:51.238984 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:51.238947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:52.219938 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:52.219889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:57.511307 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:57.511232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.040775 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.040739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"]
Sep 18 18:14:58.041008 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.040969    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" podUID="3e9a607d-3127-43ae-b18c-9dc84f52857c" containerName="maas-api" containerID="cri-o://cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0" gracePeriod=30
Sep 18 18:14:58.067597 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.067568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.104231 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.104204    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"]
Sep 18 18:14:58.104608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.104418    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc" podUID="678ace46-6812-45c2-9a74-ba9baa7c4f60" containerName="payload-pre-processing" containerID="cri-o://2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4" gracePeriod=30
Sep 18 18:14:58.132407 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.132381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.234217 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.234116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.294827 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.293587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.368942 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.368918    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:58.397858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.397834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:58.440469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.440431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.471213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prfz9\" (UniqueName: \"kubernetes.io/projected/3e9a607d-3127-43ae-b18c-9dc84f52857c-kube-api-access-prfz9\") pod \"3e9a607d-3127-43ae-b18c-9dc84f52857c\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") "
Sep 18 18:14:58.471367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-metrics-certs\") pod \"3e9a607d-3127-43ae-b18c-9dc84f52857c\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") "
Sep 18 18:14:58.471367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471317    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls\") pod \"3e9a607d-3127-43ae-b18c-9dc84f52857c\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") "
Sep 18 18:14:58.471489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a607d-3127-43ae-b18c-9dc84f52857c-openshift-service-ca\") pod \"3e9a607d-3127-43ae-b18c-9dc84f52857c\" (UID: \"3e9a607d-3127-43ae-b18c-9dc84f52857c\") "
Sep 18 18:14:58.471489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471443    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj2hg\" (UniqueName: \"kubernetes.io/projected/678ace46-6812-45c2-9a74-ba9baa7c4f60-kube-api-access-lj2hg\") pod \"678ace46-6812-45c2-9a74-ba9baa7c4f60\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") "
Sep 18 18:14:58.471489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678ace46-6812-45c2-9a74-ba9baa7c4f60-plugins-config-volume\") pod \"678ace46-6812-45c2-9a74-ba9baa7c4f60\" (UID: \"678ace46-6812-45c2-9a74-ba9baa7c4f60\") "
Sep 18 18:14:58.471840 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9a607d-3127-43ae-b18c-9dc84f52857c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e9a607d-3127-43ae-b18c-9dc84f52857c" (UID: "3e9a607d-3127-43ae-b18c-9dc84f52857c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:14:58.471966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.471846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678ace46-6812-45c2-9a74-ba9baa7c4f60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "678ace46-6812-45c2-9a74-ba9baa7c4f60" (UID: "678ace46-6812-45c2-9a74-ba9baa7c4f60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:14:58.473734 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.473712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9a607d-3127-43ae-b18c-9dc84f52857c-kube-api-access-prfz9" (OuterVolumeSpecName: "kube-api-access-prfz9") pod "3e9a607d-3127-43ae-b18c-9dc84f52857c" (UID: "3e9a607d-3127-43ae-b18c-9dc84f52857c"). InnerVolumeSpecName "kube-api-access-prfz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:14:58.473854 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.473740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678ace46-6812-45c2-9a74-ba9baa7c4f60-kube-api-access-lj2hg" (OuterVolumeSpecName: "kube-api-access-lj2hg") pod "678ace46-6812-45c2-9a74-ba9baa7c4f60" (UID: "678ace46-6812-45c2-9a74-ba9baa7c4f60"). InnerVolumeSpecName "kube-api-access-lj2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:14:58.473970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.473947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e9a607d-3127-43ae-b18c-9dc84f52857c" (UID: "3e9a607d-3127-43ae-b18c-9dc84f52857c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:14:58.473970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.473957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3e9a607d-3127-43ae-b18c-9dc84f52857c" (UID: "3e9a607d-3127-43ae-b18c-9dc84f52857c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:14:58.572805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.572720    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj2hg\" (UniqueName: \"kubernetes.io/projected/678ace46-6812-45c2-9a74-ba9baa7c4f60-kube-api-access-lj2hg\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:14:58.572805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.572750    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678ace46-6812-45c2-9a74-ba9baa7c4f60-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:14:58.572805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.572763    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prfz9\" (UniqueName: \"kubernetes.io/projected/3e9a607d-3127-43ae-b18c-9dc84f52857c-kube-api-access-prfz9\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:14:58.572805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.572776    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:14:58.572805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.572788    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e9a607d-3127-43ae-b18c-9dc84f52857c-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:14:58.572805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.572800    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a607d-3127-43ae-b18c-9dc84f52857c-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:14:58.652645 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.652616    2580 generic.go:356] "Generic (PLEG): container finished" podID="678ace46-6812-45c2-9a74-ba9baa7c4f60" containerID="2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4" exitCode=0
Sep 18 18:14:58.652802 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.652668    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"
Sep 18 18:14:58.652802 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.652714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc" event={"ID":"678ace46-6812-45c2-9a74-ba9baa7c4f60","Type":"ContainerDied","Data":"2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4"}
Sep 18 18:14:58.652802 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.652761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc" event={"ID":"678ace46-6812-45c2-9a74-ba9baa7c4f60","Type":"ContainerDied","Data":"7ecd29ad145a7bb8109475e83d9290f677e5d34ac5aec12c859a4b77ec9b9e14"}
Sep 18 18:14:58.652802 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.652786    2580 scope.go:117] "RemoveContainer" containerID="2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4"
Sep 18 18:14:58.654169 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.654139    2580 generic.go:356] "Generic (PLEG): container finished" podID="3e9a607d-3127-43ae-b18c-9dc84f52857c" containerID="cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0" exitCode=0
Sep 18 18:14:58.654297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.654175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" event={"ID":"3e9a607d-3127-43ae-b18c-9dc84f52857c","Type":"ContainerDied","Data":"cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0"}
Sep 18 18:14:58.654297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.654189    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"
Sep 18 18:14:58.654297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.654229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm" event={"ID":"3e9a607d-3127-43ae-b18c-9dc84f52857c","Type":"ContainerDied","Data":"4e534f057a5e965d7976b3a9025915af4db11e0148e1ac39daf44222bb2d2d83"}
Sep 18 18:14:58.661739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.661721    2580 scope.go:117] "RemoveContainer" containerID="2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4"
Sep 18 18:14:58.662120 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:58.662040    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4\": container with ID starting with 2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4 not found: ID does not exist" containerID="2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4"
Sep 18 18:14:58.662120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.662071    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4"} err="failed to get container status \"2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4\": rpc error: code = NotFound desc = could not find container \"2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4\": container with ID starting with 2ad6fe5305c922b0e8312471955035897aec6975774e7bec4b209c63d29667a4 not found: ID does not exist"
Sep 18 18:14:58.662120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.662091    2580 scope.go:117] "RemoveContainer" containerID="cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0"
Sep 18 18:14:58.670486 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.670466    2580 scope.go:117] "RemoveContainer" containerID="cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0"
Sep 18 18:14:58.670743 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:14:58.670724    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0\": container with ID starting with cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0 not found: ID does not exist" containerID="cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0"
Sep 18 18:14:58.670803 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.670750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0"} err="failed to get container status \"cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0\": rpc error: code = NotFound desc = could not find container \"cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0\": container with ID starting with cbf3d0560ac2618b51dbaf7da4ab36db202c42232218ebc16384a3e9bb7006c0 not found: ID does not exist"
Sep 18 18:14:58.677810 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.677786    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"]
Sep 18 18:14:58.681693 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.681673    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7fbf2b1b-657d97866-kz2rc"]
Sep 18 18:14:58.691441 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.691418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"]
Sep 18 18:14:58.699168 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.699150    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fbf2b1b-86dcf78b64-g6ghm"]
Sep 18 18:14:58.716829 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.716805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:58.933011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:58.932978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.032086 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.032053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.151963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.151935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.455322 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455290    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"]
Sep 18 18:14:59.455686 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455673    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678ace46-6812-45c2-9a74-ba9baa7c4f60" containerName="payload-pre-processing"
Sep 18 18:14:59.455753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455686    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="678ace46-6812-45c2-9a74-ba9baa7c4f60" containerName="payload-pre-processing"
Sep 18 18:14:59.455753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455706    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9a607d-3127-43ae-b18c-9dc84f52857c" containerName="maas-api"
Sep 18 18:14:59.455753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455712    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9a607d-3127-43ae-b18c-9dc84f52857c" containerName="maas-api"
Sep 18 18:14:59.455905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455779    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9a607d-3127-43ae-b18c-9dc84f52857c" containerName="maas-api"
Sep 18 18:14:59.455905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.455785    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="678ace46-6812-45c2-9a74-ba9baa7c4f60" containerName="payload-pre-processing"
Sep 18 18:14:59.464441 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.464411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.466933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.466908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2e816045-gw-openshift-default-dockercfg-k9gqd\""
Sep 18 18:14:59.469440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.469411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"]
Sep 18 18:14:59.503623 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.503594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.581903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.581872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-token\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.581910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9317f-d939-4939-82ce-71000cf5ed78-istiod-ca-cert\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.581936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-credential-socket\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.581953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-podinfo\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.581977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcqf\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-kube-api-access-pwcqf\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.582080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-socket\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.582129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-envoy\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.582172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-data\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.582292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.582227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-certs\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.603433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.603410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.683716 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-token\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.683839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9317f-d939-4939-82ce-71000cf5ed78-istiod-ca-cert\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.683839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-credential-socket\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.683839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-podinfo\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.683839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcqf\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-kube-api-access-pwcqf\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-socket\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-envoy\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-data\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.683962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-certs\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.684344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-certs\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.684418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-credential-socket\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.684450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-socket\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.684558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-data\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.684692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.684675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9317f-d939-4939-82ce-71000cf5ed78-istiod-ca-cert\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.686681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.686657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-envoy\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.686950 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.686928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-podinfo\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.691465 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.691440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-token\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.691600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.691582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcqf\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-kube-api-access-pwcqf\") pod \"e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") " pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.776945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.776876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:14:59.856978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.856940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.910449 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.910422    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9a607d-3127-43ae-b18c-9dc84f52857c" path="/var/lib/kubelet/pods/3e9a607d-3127-43ae-b18c-9dc84f52857c/volumes"
Sep 18 18:14:59.910773 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:14:59.910760    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="678ace46-6812-45c2-9a74-ba9baa7c4f60" path="/var/lib/kubelet/pods/678ace46-6812-45c2-9a74-ba9baa7c4f60/volumes"
Sep 18 18:15:00.073567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.073545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"]
Sep 18 18:15:00.075811 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:15:00.075786    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d9317f_d939_4939_82ce_71000cf5ed78.slice/crio-fb64e10b99f2ec5395b5a9d4f5f71f098a695476fe66f86b9793d36e55a5fd87 WatchSource:0}: Error finding container fb64e10b99f2ec5395b5a9d4f5f71f098a695476fe66f86b9793d36e55a5fd87: Status 404 returned error can't find the container with id fb64e10b99f2ec5395b5a9d4f5f71f098a695476fe66f86b9793d36e55a5fd87
Sep 18 18:15:00.078642 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.078612    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:00.078728 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.078673    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:00.078728 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.078715    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:00.277177 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.277147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:00.667842 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.667811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" event={"ID":"f8d9317f-d939-4939-82ce-71000cf5ed78","Type":"ContainerStarted","Data":"5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"}
Sep 18 18:15:00.668333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.667872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" event={"ID":"f8d9317f-d939-4939-82ce-71000cf5ed78","Type":"ContainerStarted","Data":"fb64e10b99f2ec5395b5a9d4f5f71f098a695476fe66f86b9793d36e55a5fd87"}
Sep 18 18:15:00.686388 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.686343    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podStartSLOduration=1.686328501 podStartE2EDuration="1.686328501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:00.684593947 +0000 UTC m=+870.836468783" watchObservedRunningTime="2026-09-18 18:15:00.686328501 +0000 UTC m=+870.838203360"
Sep 18 18:15:00.777906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:00.777877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:01.127346 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:01.127241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:01.782120 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:01.782092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:01.893552 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:01.893521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:02.217227 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.217201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:02.402997 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.402959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:02.672186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.672153    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"]
Sep 18 18:15:02.677280 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.677259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.679466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.679445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2e816045-serving-cert\""
Sep 18 18:15:02.680083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.680066    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:02.681277 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.681259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:02.689942 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.689920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"]
Sep 18 18:15:02.703489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.703463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:02.802825 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.802795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:02.813358 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.813321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-metrics-certs\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.813499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.813379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859c1531-564b-435e-b5eb-447d8aeaca45-openshift-service-ca\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.813499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.813436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.813499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.813458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdm6\" (UniqueName: \"kubernetes.io/projected/859c1531-564b-435e-b5eb-447d8aeaca45-kube-api-access-mhdm6\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.914559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.914529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-metrics-certs\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.914733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.914567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859c1531-564b-435e-b5eb-447d8aeaca45-openshift-service-ca\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.914733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.914617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.914733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.914640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdm6\" (UniqueName: \"kubernetes.io/projected/859c1531-564b-435e-b5eb-447d8aeaca45-kube-api-access-mhdm6\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.914867 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:02.914755    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-serving-cert: secret "maas-api-e2e-migrate-2e816045-serving-cert" not found
Sep 18 18:15:02.914867 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:02.914835    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls podName:859c1531-564b-435e-b5eb-447d8aeaca45 nodeName:}" failed. No retries permitted until 2026-09-18 18:15:03.414812271 +0000 UTC m=+873.566687097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls") pod "maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" (UID: "859c1531-564b-435e-b5eb-447d8aeaca45") : secret "maas-api-e2e-migrate-2e816045-serving-cert" not found
Sep 18 18:15:02.915499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.915478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859c1531-564b-435e-b5eb-447d8aeaca45-openshift-service-ca\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.917275 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.917260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-metrics-certs\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:02.922548 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:02.922491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdm6\" (UniqueName: \"kubernetes.io/projected/859c1531-564b-435e-b5eb-447d8aeaca45-kube-api-access-mhdm6\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:03.199277 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:03.199182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:03.293460 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:03.293425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:03.420580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:03.420546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:03.420873 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:03.420689    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-serving-cert: secret "maas-api-e2e-migrate-2e816045-serving-cert" not found
Sep 18 18:15:03.420873 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:03.420747    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls podName:859c1531-564b-435e-b5eb-447d8aeaca45 nodeName:}" failed. No retries permitted until 2026-09-18 18:15:04.420732812 +0000 UTC m=+874.572607629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls") pod "maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" (UID: "859c1531-564b-435e-b5eb-447d8aeaca45") : secret "maas-api-e2e-migrate-2e816045-serving-cert" not found
Sep 18 18:15:03.612140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:03.612049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:03.704329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:03.704292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:04.207753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:04.207717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:04.429855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:04.429811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:04.432482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:04.432464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") pod \"maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:04.490488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:04.490421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:04.764590 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:04.764566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"]
Sep 18 18:15:04.767222 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:15:04.767194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859c1531_564b_435e_b5eb_447d8aeaca45.slice/crio-3c111dcc3321eefcfdf2db6e2834804987204572dee9bf8014b1a640570dd714 WatchSource:0}: Error finding container 3c111dcc3321eefcfdf2db6e2834804987204572dee9bf8014b1a640570dd714: Status 404 returned error can't find the container with id 3c111dcc3321eefcfdf2db6e2834804987204572dee9bf8014b1a640570dd714
Sep 18 18:15:05.019400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:05.019326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:05.652376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:05.652342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:05.695667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:05.695603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" event={"ID":"859c1531-564b-435e-b5eb-447d8aeaca45","Type":"ContainerStarted","Data":"d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7"}
Sep 18 18:15:05.695667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:05.695648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" event={"ID":"859c1531-564b-435e-b5eb-447d8aeaca45","Type":"ContainerStarted","Data":"3c111dcc3321eefcfdf2db6e2834804987204572dee9bf8014b1a640570dd714"}
Sep 18 18:15:05.695848 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:05.695720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:05.712641 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:05.712602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" podStartSLOduration=3.3334517200000002 podStartE2EDuration="3.712589203s" totalImagesPullingTime="379.137483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:05.711486697 +0000 UTC m=+875.863361532" watchObservedRunningTime="2026-09-18 18:15:05.712589203 +0000 UTC m=+875.864464038"
Sep 18 18:15:06.365359 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:06.365318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:08.888614 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:08.888582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:11.706153 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:11.706124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:22.787131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:22.787097    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerID="5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40" exitCode=0
Sep 18 18:15:22.787532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:22.787143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" event={"ID":"f8d9317f-d939-4939-82ce-71000cf5ed78","Type":"ContainerDied","Data":"5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"}
Sep 18 18:15:22.787575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:22.787536    2580 scope.go:117] "RemoveContainer" containerID="5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"
Sep 18 18:15:22.788663 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:22.788626    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:22.788761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:22.788716    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:22.788823 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:22.788764    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:23.794147 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:23.794102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" event={"ID":"f8d9317f-d939-4939-82ce-71000cf5ed78","Type":"ContainerStarted","Data":"7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9"}
Sep 18 18:15:24.683311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:24.683269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:24.777619 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:24.777586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:24.777857 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:24.777834    2580 patch_prober.go:28] interesting pod/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 18 18:15:24.777920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:24.777887    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 18 18:15:25.778150 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:25.778115    2580 patch_prober.go:28] interesting pod/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 18 18:15:25.778526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:25.778167    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 18 18:15:26.007620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:26.007584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:26.121664 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:26.121535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:26.778130 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:26.778094    2580 patch_prober.go:28] interesting pod/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 18 18:15:26.778300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:26.778164    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 18 18:15:26.908840 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:26.908796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:27.711438 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:27.711400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:27.778147 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:27.778106    2580 patch_prober.go:28] interesting pod/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 18 18:15:27.778320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:27.778181    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 18 18:15:28.416190 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:28.416149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:28.777944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:28.777857    2580 patch_prober.go:28] interesting pod/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 18 18:15:28.778133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:28.777952    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 18 18:15:29.308766 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:29.308732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:29.781437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:29.781411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:29.781804 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:29.781777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:29.782537 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:29.782518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:29.883413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:29.883382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:30.414186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:30.414153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:30.530685 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:30.530654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:31.319186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:31.319151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:31.604404 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:31.604329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:33.316782 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:33.316711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:33.858088 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:33.858055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"]
Sep 18 18:15:33.858288 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:33.858264    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" podUID="859c1531-564b-435e-b5eb-447d8aeaca45" containerName="maas-api" containerID="cri-o://d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7" gracePeriod=30
Sep 18 18:15:33.885909 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:33.885886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:33.986740 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:33.986709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.086968 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.086929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.112147 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.112097    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:34.187084 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.187056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.212137 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.212108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-metrics-certs\") pod \"859c1531-564b-435e-b5eb-447d8aeaca45\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") "
Sep 18 18:15:34.212271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.212149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859c1531-564b-435e-b5eb-447d8aeaca45-openshift-service-ca\") pod \"859c1531-564b-435e-b5eb-447d8aeaca45\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") "
Sep 18 18:15:34.212271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.212214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhdm6\" (UniqueName: \"kubernetes.io/projected/859c1531-564b-435e-b5eb-447d8aeaca45-kube-api-access-mhdm6\") pod \"859c1531-564b-435e-b5eb-447d8aeaca45\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") "
Sep 18 18:15:34.212271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.212246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") pod \"859c1531-564b-435e-b5eb-447d8aeaca45\" (UID: \"859c1531-564b-435e-b5eb-447d8aeaca45\") "
Sep 18 18:15:34.212617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.212588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859c1531-564b-435e-b5eb-447d8aeaca45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "859c1531-564b-435e-b5eb-447d8aeaca45" (UID: "859c1531-564b-435e-b5eb-447d8aeaca45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:15:34.214561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.214534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "859c1531-564b-435e-b5eb-447d8aeaca45" (UID: "859c1531-564b-435e-b5eb-447d8aeaca45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:15:34.214902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.214882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "859c1531-564b-435e-b5eb-447d8aeaca45" (UID: "859c1531-564b-435e-b5eb-447d8aeaca45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:15:34.214902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.214894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859c1531-564b-435e-b5eb-447d8aeaca45-kube-api-access-mhdm6" (OuterVolumeSpecName: "kube-api-access-mhdm6") pod "859c1531-564b-435e-b5eb-447d8aeaca45" (UID: "859c1531-564b-435e-b5eb-447d8aeaca45"). InnerVolumeSpecName "kube-api-access-mhdm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:15:34.245731 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.245702    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"]
Sep 18 18:15:34.245919 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.245895    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy" containerID="cri-o://7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9" gracePeriod=30
Sep 18 18:15:34.269869 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.269841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.313639 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.313615    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhdm6\" (UniqueName: \"kubernetes.io/projected/859c1531-564b-435e-b5eb-447d8aeaca45-kube-api-access-mhdm6\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:34.313639 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.313637    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:34.313786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.313648    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859c1531-564b-435e-b5eb-447d8aeaca45-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:34.313786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.313656    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859c1531-564b-435e-b5eb-447d8aeaca45-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:34.442626 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.442579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.743431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.743330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.849624 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.849589    2580 generic.go:356] "Generic (PLEG): container finished" podID="859c1531-564b-435e-b5eb-447d8aeaca45" containerID="d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7" exitCode=0
Sep 18 18:15:34.849799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.849635    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"
Sep 18 18:15:34.849799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.849676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" event={"ID":"859c1531-564b-435e-b5eb-447d8aeaca45","Type":"ContainerDied","Data":"d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7"}
Sep 18 18:15:34.849799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.849743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk" event={"ID":"859c1531-564b-435e-b5eb-447d8aeaca45","Type":"ContainerDied","Data":"3c111dcc3321eefcfdf2db6e2834804987204572dee9bf8014b1a640570dd714"}
Sep 18 18:15:34.849799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.849782    2580 scope.go:117] "RemoveContainer" containerID="d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7"
Sep 18 18:15:34.859531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.859385    2580 scope.go:117] "RemoveContainer" containerID="d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7"
Sep 18 18:15:34.859622 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:34.859607    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7\": container with ID starting with d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7 not found: ID does not exist" containerID="d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7"
Sep 18 18:15:34.859663 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.859630    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7"} err="failed to get container status \"d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7\": rpc error: code = NotFound desc = could not find container \"d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7\": container with ID starting with d6fedee5a6847b25ffbf6d24df8cd25475a77373b4ed3c0e7c1d0b3f32c3e9f7 not found: ID does not exist"
Sep 18 18:15:34.882219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.882199    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"]
Sep 18 18:15:34.885908 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:34.885888    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2e816045-66886bfb5d-fzhtk"]
Sep 18 18:15:35.034683 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.034601    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"]
Sep 18 18:15:35.035015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.035001    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859c1531-564b-435e-b5eb-447d8aeaca45" containerName="maas-api"
Sep 18 18:15:35.035015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.035016    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="859c1531-564b-435e-b5eb-447d8aeaca45" containerName="maas-api"
Sep 18 18:15:35.035165 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.035109    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="859c1531-564b-435e-b5eb-447d8aeaca45" containerName="maas-api"
Sep 18 18:15:35.039839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.039817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.042615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.042592    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89dea9e1-openshift-default-dockercfg-mncsn\""
Sep 18 18:15:35.059902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.059871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"]
Sep 18 18:15:35.068174 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.068140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:35.120611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-istio-token\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-certs\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38163c0f-923c-484f-a54a-191c0ca724a3-istio-podinfo\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8vz\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-kube-api-access-hb8vz\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-socket\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38163c0f-923c-484f-a54a-191c0ca724a3-istiod-ca-cert\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-envoy\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120995 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-data\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.120995 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.120921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-credential-socket\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.165319 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.165283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:35.222063 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-data\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-credential-socket\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-istio-token\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-certs\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38163c0f-923c-484f-a54a-191c0ca724a3-istio-podinfo\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8vz\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-kube-api-access-hb8vz\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-socket\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38163c0f-923c-484f-a54a-191c0ca724a3-istiod-ca-cert\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-envoy\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-data\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-credential-socket\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-socket\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.222815 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.222794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-certs\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.223517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.223444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38163c0f-923c-484f-a54a-191c0ca724a3-istiod-ca-cert\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.225898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.225858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-envoy\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.226243 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.226218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38163c0f-923c-484f-a54a-191c0ca724a3-istio-podinfo\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.232288 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.232267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-istio-token\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.233134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.233107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8vz\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-kube-api-access-hb8vz\") pod \"e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") " pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.352922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.352843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:35.403071 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.402925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:35.631431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.631406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"]
Sep 18 18:15:35.633165 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:15:35.633134    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38163c0f_923c_484f_a54a_191c0ca724a3.slice/crio-5ce9baa8dbea40c01b9df7b538aa941b89a45a4dcc70651ce7964e01dc53e3c6 WatchSource:0}: Error finding container 5ce9baa8dbea40c01b9df7b538aa941b89a45a4dcc70651ce7964e01dc53e3c6: Status 404 returned error can't find the container with id 5ce9baa8dbea40c01b9df7b538aa941b89a45a4dcc70651ce7964e01dc53e3c6
Sep 18 18:15:35.635462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.635428    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:35.635546 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.635492    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:35.635546 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.635519    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:35.857515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.857473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" event={"ID":"38163c0f-923c-484f-a54a-191c0ca724a3","Type":"ContainerStarted","Data":"835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"}
Sep 18 18:15:35.857659 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.857526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" event={"ID":"38163c0f-923c-484f-a54a-191c0ca724a3","Type":"ContainerStarted","Data":"5ce9baa8dbea40c01b9df7b538aa941b89a45a4dcc70651ce7964e01dc53e3c6"}
Sep 18 18:15:35.899324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.899224    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podStartSLOduration=0.899205108 podStartE2EDuration="899.205108ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:35.897565098 +0000 UTC m=+906.049439933" watchObservedRunningTime="2026-09-18 18:15:35.899205108 +0000 UTC m=+906.051079945"
Sep 18 18:15:35.911890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.911864    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859c1531-564b-435e-b5eb-447d8aeaca45" path="/var/lib/kubelet/pods/859c1531-564b-435e-b5eb-447d8aeaca45/volumes"
Sep 18 18:15:35.956049 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:35.955996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:36.353091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:36.353059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:36.358347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:36.358324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:36.728906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:36.728862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:36.863041 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:36.863005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:36.863962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:36.863937    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:15:37.250425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:37.250385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:37.340383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:37.340346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:37.714782 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:37.714741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:38.434995 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:38.434948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:38.532065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:38.532009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:38.922649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:38.922613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:39.105062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.105012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:39.209826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.209737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:39.210504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.210484    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"]
Sep 18 18:15:39.214151 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.214136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.216209 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.216187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89dea9e1\""
Sep 18 18:15:39.216324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.216306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89dea9e1-dockercfg-ckfng\""
Sep 18 18:15:39.227237 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.227173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"]
Sep 18 18:15:39.363810 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.363775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd9deca-585a-445a-ac23-1b2d59c28fb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.363977 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.363887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxsd9\" (UniqueName: \"kubernetes.io/projected/ecd9deca-585a-445a-ac23-1b2d59c28fb8-kube-api-access-lxsd9\") pod \"payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.464710 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.464629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxsd9\" (UniqueName: \"kubernetes.io/projected/ecd9deca-585a-445a-ac23-1b2d59c28fb8-kube-api-access-lxsd9\") pod \"payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.465152 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.464779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd9deca-585a-445a-ac23-1b2d59c28fb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.465586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.465561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd9deca-585a-445a-ac23-1b2d59c28fb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.472628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.472606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxsd9\" (UniqueName: \"kubernetes.io/projected/ecd9deca-585a-445a-ac23-1b2d59c28fb8-kube-api-access-lxsd9\") pod \"payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.499897 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.499865    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:39.528875 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.528535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.565845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.565770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-certs\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.566348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.566190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:15:39.566348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.566271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-envoy\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.566815 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.566794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-socket\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.567074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.566849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-data\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.567216 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.567185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:15:39.567764 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.567737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:15:39.567859 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.567811    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-podinfo\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.567859 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.567853    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-credential-socket\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.567968 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.567899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwcqf\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-kube-api-access-pwcqf\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.568536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.568509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:15:39.568920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.568894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d9317f-d939-4939-82ce-71000cf5ed78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:15:39.569006 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.568973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9317f-d939-4939-82ce-71000cf5ed78-istiod-ca-cert\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.569094 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.569018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-token\") pod \"f8d9317f-d939-4939-82ce-71000cf5ed78\" (UID: \"f8d9317f-d939-4939-82ce-71000cf5ed78\") "
Sep 18 18:15:39.569402 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.569379    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.569476 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.569410    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.569476 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.569425    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.569476 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.569439    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9317f-d939-4939-82ce-71000cf5ed78-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.569476 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.569468    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.571195 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.570873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:15:39.571617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.571590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-kube-api-access-pwcqf" (OuterVolumeSpecName: "kube-api-access-pwcqf") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "kube-api-access-pwcqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:15:39.571708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.571656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:15:39.572645 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.572619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8d9317f-d939-4939-82ce-71000cf5ed78" (UID: "f8d9317f-d939-4939-82ce-71000cf5ed78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:15:39.669974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.669947    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.669974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.669972    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwcqf\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-kube-api-access-pwcqf\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.669974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.669981    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.670220 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.669990    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d9317f-d939-4939-82ce-71000cf5ed78-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:15:39.705740 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.705693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:39.813972 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.813941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"]
Sep 18 18:15:39.815915 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:15:39.815887    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd9deca_585a_445a_ac23_1b2d59c28fb8.slice/crio-b5b21ee92c3cb0ce7117b74f4411eafea4a93b6a581e1477787591089cc8eabb WatchSource:0}: Error finding container b5b21ee92c3cb0ce7117b74f4411eafea4a93b6a581e1477787591089cc8eabb: Status 404 returned error can't find the container with id b5b21ee92c3cb0ce7117b74f4411eafea4a93b6a581e1477787591089cc8eabb
Sep 18 18:15:39.879164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.879130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84" event={"ID":"ecd9deca-585a-445a-ac23-1b2d59c28fb8","Type":"ContainerStarted","Data":"fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956"}
Sep 18 18:15:39.879278 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.879196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84" event={"ID":"ecd9deca-585a-445a-ac23-1b2d59c28fb8","Type":"ContainerStarted","Data":"b5b21ee92c3cb0ce7117b74f4411eafea4a93b6a581e1477787591089cc8eabb"}
Sep 18 18:15:39.879278 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.879224    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:39.880931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.880906    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerID="7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9" exitCode=0
Sep 18 18:15:39.881039 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.880953    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"
Sep 18 18:15:39.881039 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.880988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" event={"ID":"f8d9317f-d939-4939-82ce-71000cf5ed78","Type":"ContainerDied","Data":"7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9"}
Sep 18 18:15:39.881125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.881055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l" event={"ID":"f8d9317f-d939-4939-82ce-71000cf5ed78","Type":"ContainerDied","Data":"fb64e10b99f2ec5395b5a9d4f5f71f098a695476fe66f86b9793d36e55a5fd87"}
Sep 18 18:15:39.881125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.881083    2580 scope.go:117] "RemoveContainer" containerID="7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9"
Sep 18 18:15:39.890458 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.890442    2580 scope.go:117] "RemoveContainer" containerID="5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"
Sep 18 18:15:39.897370 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.897329    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84" podStartSLOduration=0.897316074 podStartE2EDuration="897.316074ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:39.896709512 +0000 UTC m=+910.048584347" watchObservedRunningTime="2026-09-18 18:15:39.897316074 +0000 UTC m=+910.049190910"
Sep 18 18:15:39.899851 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.899829    2580 scope.go:117] "RemoveContainer" containerID="7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9"
Sep 18 18:15:39.900188 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:39.900167    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9\": container with ID starting with 7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9 not found: ID does not exist" containerID="7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9"
Sep 18 18:15:39.900254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.900195    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9"} err="failed to get container status \"7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9\": rpc error: code = NotFound desc = could not find container \"7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9\": container with ID starting with 7259d4ffbfc6824f6d59c7d2bb7744c8e911d82a3c59cb5376e184147091a2e9 not found: ID does not exist"
Sep 18 18:15:39.900254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.900213    2580 scope.go:117] "RemoveContainer" containerID="5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"
Sep 18 18:15:39.900515 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:15:39.900490    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40\": container with ID starting with 5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40 not found: ID does not exist" containerID="5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"
Sep 18 18:15:39.900600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.900520    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40"} err="failed to get container status \"5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40\": rpc error: code = NotFound desc = could not find container \"5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40\": container with ID starting with 5c7d948e958ff3b9b113e830d20005725f28039d69be35cb0e68b0bc012b0a40 not found: ID does not exist"
Sep 18 18:15:39.913795 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.913765    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"]
Sep 18 18:15:39.917942 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:39.917923    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2e816045-gw-openshift-default-7c46c9448d-hbq8l"]
Sep 18 18:15:40.324209 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:40.324175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:41.009647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:41.009609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:41.821282 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:41.821250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:41.910696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:41.910666    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" path="/var/lib/kubelet/pods/f8d9317f-d939-4939-82ce-71000cf5ed78/volumes"
Sep 18 18:15:50.889166 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:50.889132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:15:53.509629 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.509576    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"]
Sep 18 18:15:53.510202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.510180    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy"
Sep 18 18:15:53.510290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.510205    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy"
Sep 18 18:15:53.510290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.510218    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy"
Sep 18 18:15:53.510290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.510226    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy"
Sep 18 18:15:53.510399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.510297    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy"
Sep 18 18:15:53.510399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.510308    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d9317f-d939-4939-82ce-71000cf5ed78" containerName="istio-proxy"
Sep 18 18:15:53.513324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.513304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.515537 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.515515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-de7535\"/\"openshift-service-ca.crt\""
Sep 18 18:15:53.515646 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.515520    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-de7535\"/\"test-model-de7535-kserve-self-signed-certs\""
Sep 18 18:15:53.516472 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.516267    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-de7535\"/\"default-dockercfg-qxdxk\""
Sep 18 18:15:53.516472 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.516305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-de7535\"/\"kube-root-ca.crt\""
Sep 18 18:15:53.529050 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.523282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"]
Sep 18 18:15:53.539049 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.534737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:53.611294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.611267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-tmp-dir\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.611424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.611330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-home\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.611424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.611374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-dshm\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.611424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.611398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbgr\" (UniqueName: \"kubernetes.io/projected/d2e0c651-dfea-4c75-a64c-f0815b86f287-kube-api-access-zlbgr\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.611558 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.611442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-model-cache\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.611558 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.611462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0c651-dfea-4c75-a64c-f0815b86f287-tls-certs\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.711854 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.711820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-home\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.711863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-dshm\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.711955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbgr\" (UniqueName: \"kubernetes.io/projected/d2e0c651-dfea-4c75-a64c-f0815b86f287-kube-api-access-zlbgr\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.712000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-model-cache\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.712061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0c651-dfea-4c75-a64c-f0815b86f287-tls-certs\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.712144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-tmp-dir\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.712300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-home\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712430 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.712404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-model-cache\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.712490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.712460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-tmp-dir\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.714469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.714447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-dshm\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.714723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.714703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0c651-dfea-4c75-a64c-f0815b86f287-tls-certs\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.719582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.719560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbgr\" (UniqueName: \"kubernetes.io/projected/d2e0c651-dfea-4c75-a64c-f0815b86f287-kube-api-access-zlbgr\") pod \"test-model-de7535-kserve-7cb58cbfb7-4cjmt\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") " pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:53.824343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:53.824006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:15:54.115351 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:54.115322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"]
Sep 18 18:15:54.117209 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:15:54.117182    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2e0c651_dfea_4c75_a64c_f0815b86f287.slice/crio-25e2a6ef298eac948d9fe140a0c4596845cc5133b763dbd2ca3109d20d389f2c WatchSource:0}: Error finding container 25e2a6ef298eac948d9fe140a0c4596845cc5133b763dbd2ca3109d20d389f2c: Status 404 returned error can't find the container with id 25e2a6ef298eac948d9fe140a0c4596845cc5133b763dbd2ca3109d20d389f2c
Sep 18 18:15:54.977104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:54.977062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt" event={"ID":"d2e0c651-dfea-4c75-a64c-f0815b86f287","Type":"ContainerStarted","Data":"7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7"}
Sep 18 18:15:54.977473 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:54.977113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt" event={"ID":"d2e0c651-dfea-4c75-a64c-f0815b86f287","Type":"ContainerStarted","Data":"25e2a6ef298eac948d9fe140a0c4596845cc5133b763dbd2ca3109d20d389f2c"}
Sep 18 18:15:54.996087 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:54.996021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt" podStartSLOduration=1.996003044 podStartE2EDuration="1.996003044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:54.993491553 +0000 UTC m=+925.145366388" watchObservedRunningTime="2026-09-18 18:15:54.996003044 +0000 UTC m=+925.147877887"
Sep 18 18:15:56.913198 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:56.913164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:59.001808 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:59.001766    2580 generic.go:356] "Generic (PLEG): container finished" podID="38163c0f-923c-484f-a54a-191c0ca724a3" containerID="835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c" exitCode=0
Sep 18 18:15:59.002352 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:59.001843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" event={"ID":"38163c0f-923c-484f-a54a-191c0ca724a3","Type":"ContainerDied","Data":"835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"}
Sep 18 18:15:59.002466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:59.002354    2580 scope.go:117] "RemoveContainer" containerID="835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"
Sep 18 18:15:59.003480 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:59.003450    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:59.003559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:59.003518    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:15:59.003559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:15:59.003551    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:16:00.008433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:00.008399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" event={"ID":"38163c0f-923c-484f-a54a-191c0ca724a3","Type":"ContainerStarted","Data":"5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e"}
Sep 18 18:16:00.353307 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:00.353226    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:16:00.353457 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:00.353412    2580 patch_prober.go:28] interesting pod/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 18 18:16:00.353502 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:00.353456    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 18 18:16:00.516656 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:00.516613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:01.307320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:01.307274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:01.354258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:01.354219    2580 patch_prober.go:28] interesting pod/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 18 18:16:01.354453 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:01.354283    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 18 18:16:01.846195 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:01.846154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:02.354047 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:02.353990    2580 patch_prober.go:28] interesting pod/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 18 18:16:02.354549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:02.354097    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 18 18:16:02.509453 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:02.509424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:03.351722 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.351690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:03.353587 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.353561    2580 patch_prober.go:28] interesting pod/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 18 18:16:03.353853 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.353817    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 18 18:16:03.812377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.812339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:03.825922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.825207    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:16:03.825922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.825256    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:16:03.861428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:03.861404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:16:04.060876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:04.060850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:16:04.109837 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:04.109732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:04.353756 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:04.353708    2580 patch_prober.go:28] interesting pod/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 18 18:16:04.353962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:04.353780    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 18 18:16:04.889016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:04.888983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:05.353380 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:05.353341    2580 patch_prober.go:28] interesting pod/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 18 18:16:05.353556 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:05.353402    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 18 18:16:05.568776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:05.568736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:06.112342 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:06.112305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:06.357547 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:06.357520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:16:06.357942 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:06.357924    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:16:06.358691 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:06.358672    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:16:06.833132 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:06.833098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:07.599836 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:07.599802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:08.812905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:08.812868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:09.418586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.418542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:09.454148 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.454116    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"]
Sep 18 18:16:09.454405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.454381    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84" podUID="ecd9deca-585a-445a-ac23-1b2d59c28fb8" containerName="payload-pre-processing" containerID="cri-o://fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956" gracePeriod=30
Sep 18 18:16:09.518002 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.517970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:09.617284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.617244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:09.681000 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.680925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:09.716782 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.716756    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:16:09.769209 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.769174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxsd9\" (UniqueName: \"kubernetes.io/projected/ecd9deca-585a-445a-ac23-1b2d59c28fb8-kube-api-access-lxsd9\") pod \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") "
Sep 18 18:16:09.769345 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.769214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd9deca-585a-445a-ac23-1b2d59c28fb8-plugins-config-volume\") pod \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\" (UID: \"ecd9deca-585a-445a-ac23-1b2d59c28fb8\") "
Sep 18 18:16:09.769616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.769585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd9deca-585a-445a-ac23-1b2d59c28fb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecd9deca-585a-445a-ac23-1b2d59c28fb8" (UID: "ecd9deca-585a-445a-ac23-1b2d59c28fb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:16:09.771435 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.771415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd9deca-585a-445a-ac23-1b2d59c28fb8-kube-api-access-lxsd9" (OuterVolumeSpecName: "kube-api-access-lxsd9") pod "ecd9deca-585a-445a-ac23-1b2d59c28fb8" (UID: "ecd9deca-585a-445a-ac23-1b2d59c28fb8"). InnerVolumeSpecName "kube-api-access-lxsd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:16:09.870683 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.870639    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxsd9\" (UniqueName: \"kubernetes.io/projected/ecd9deca-585a-445a-ac23-1b2d59c28fb8-kube-api-access-lxsd9\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:09.870683 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:09.870674    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd9deca-585a-445a-ac23-1b2d59c28fb8-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:10.023582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.023485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:10.065941 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.065907    2580 generic.go:356] "Generic (PLEG): container finished" podID="ecd9deca-585a-445a-ac23-1b2d59c28fb8" containerID="fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956" exitCode=0
Sep 18 18:16:10.066113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.065952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"
Sep 18 18:16:10.066113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.065953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84" event={"ID":"ecd9deca-585a-445a-ac23-1b2d59c28fb8","Type":"ContainerDied","Data":"fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956"}
Sep 18 18:16:10.066113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.066090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84" event={"ID":"ecd9deca-585a-445a-ac23-1b2d59c28fb8","Type":"ContainerDied","Data":"b5b21ee92c3cb0ce7117b74f4411eafea4a93b6a581e1477787591089cc8eabb"}
Sep 18 18:16:10.066226 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.066119    2580 scope.go:117] "RemoveContainer" containerID="fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956"
Sep 18 18:16:10.075182 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.075166    2580 scope.go:117] "RemoveContainer" containerID="fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956"
Sep 18 18:16:10.075433 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:16:10.075410    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956\": container with ID starting with fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956 not found: ID does not exist" containerID="fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956"
Sep 18 18:16:10.075504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.075441    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956"} err="failed to get container status \"fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956\": rpc error: code = NotFound desc = could not find container \"fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956\": container with ID starting with fe8877c5cb725e57c63fd5d7a532cd3249e81c3b16bca5c0a47df25ea6283956 not found: ID does not exist"
Sep 18 18:16:10.081104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.081082    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"]
Sep 18 18:16:10.086355 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.086335    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89dea9e1-557b58f446-9ct84"]
Sep 18 18:16:10.728738 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.728705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:10.846604 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.846569    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"]
Sep 18 18:16:10.846778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.846752    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy" containerID="cri-o://5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e" gracePeriod=30
Sep 18 18:16:10.867241 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:10.867153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:11.312540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:11.312497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:11.412099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:11.412066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:11.619012 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:11.618928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:11.910932 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:11.910903    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd9deca-585a-445a-ac23-1b2d59c28fb8" path="/var/lib/kubelet/pods/ecd9deca-585a-445a-ac23-1b2d59c28fb8/volumes"
Sep 18 18:16:12.160104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:12.160068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.264417 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:12.264331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.893065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:12.893009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:13.885135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:13.885096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:14.609185 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:14.609119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:14.701139 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:14.701104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:15.325681 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:15.325641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:16.090306 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.090279    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:16:16.099671 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.099642    2580 generic.go:356] "Generic (PLEG): container finished" podID="38163c0f-923c-484f-a54a-191c0ca724a3" containerID="5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e" exitCode=0
Sep 18 18:16:16.099769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.099684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"
Sep 18 18:16:16.099769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.099695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" event={"ID":"38163c0f-923c-484f-a54a-191c0ca724a3","Type":"ContainerDied","Data":"5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e"}
Sep 18 18:16:16.099769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.099745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg" event={"ID":"38163c0f-923c-484f-a54a-191c0ca724a3","Type":"ContainerDied","Data":"5ce9baa8dbea40c01b9df7b538aa941b89a45a4dcc70651ce7964e01dc53e3c6"}
Sep 18 18:16:16.099878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.099775    2580 scope.go:117] "RemoveContainer" containerID="5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e"
Sep 18 18:16:16.108080 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.108060    2580 scope.go:117] "RemoveContainer" containerID="835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"
Sep 18 18:16:16.114695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.114669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:16.117969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.117948    2580 scope.go:117] "RemoveContainer" containerID="5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e"
Sep 18 18:16:16.118314 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:16:16.118293    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e\": container with ID starting with 5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e not found: ID does not exist" containerID="5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e"
Sep 18 18:16:16.118400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.118323    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e"} err="failed to get container status \"5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e\": rpc error: code = NotFound desc = could not find container \"5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e\": container with ID starting with 5b28b0af97b96a7f096323b167cd1694f061868d7680e71602d17f11d6b8309e not found: ID does not exist"
Sep 18 18:16:16.118400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.118342    2580 scope.go:117] "RemoveContainer" containerID="835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"
Sep 18 18:16:16.118657 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:16:16.118632    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c\": container with ID starting with 835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c not found: ID does not exist" containerID="835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"
Sep 18 18:16:16.118709 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.118667    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c"} err="failed to get container status \"835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c\": rpc error: code = NotFound desc = could not find container \"835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c\": container with ID starting with 835b5227af82ab402b6269832c388c89d8e8ab16b7e52b7dbb2be1fec97acf7c not found: ID does not exist"
Sep 18 18:16:16.234895 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.234865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-envoy\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.234903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-data\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.234924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-certs\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.234958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb8vz\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-kube-api-access-hb8vz\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.234987    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-socket\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235279 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-istio-token\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235279 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-credential-socket\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235279 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:16:16.235436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235285    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:16:16.235436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38163c0f-923c-484f-a54a-191c0ca724a3-istio-podinfo\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:16:16.235436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38163c0f-923c-484f-a54a-191c0ca724a3-istiod-ca-cert\") pod \"38163c0f-923c-484f-a54a-191c0ca724a3\" (UID: \"38163c0f-923c-484f-a54a-191c0ca724a3\") "
Sep 18 18:16:16.235640 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:16:16.235691 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235659    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.235691 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235681    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.235797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235695    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.235797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38163c0f-923c-484f-a54a-191c0ca724a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:16:16.235797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.235710    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.237535 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.237509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38163c0f-923c-484f-a54a-191c0ca724a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:16:16.237859 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.237833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-kube-api-access-hb8vz" (OuterVolumeSpecName: "kube-api-access-hb8vz") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "kube-api-access-hb8vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:16:16.238098 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.238073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:16:16.238175 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.238075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38163c0f-923c-484f-a54a-191c0ca724a3" (UID: "38163c0f-923c-484f-a54a-191c0ca724a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:16:16.336861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.336830    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38163c0f-923c-484f-a54a-191c0ca724a3-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.336861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.336855    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38163c0f-923c-484f-a54a-191c0ca724a3-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.336861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.336864    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38163c0f-923c-484f-a54a-191c0ca724a3-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.337152 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.336873    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8vz\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-kube-api-access-hb8vz\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.337152 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.336881    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38163c0f-923c-484f-a54a-191c0ca724a3-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:16.419910 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.419882    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"]
Sep 18 18:16:16.423883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.423849    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89dea9e1-openshift-default-bc5c49556-tbhjg"]
Sep 18 18:16:16.778048 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:16.777938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:17.605450 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605418    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"]
Sep 18 18:16:17.605850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605820    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd9deca-585a-445a-ac23-1b2d59c28fb8" containerName="payload-pre-processing"
Sep 18 18:16:17.605850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605834    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd9deca-585a-445a-ac23-1b2d59c28fb8" containerName="payload-pre-processing"
Sep 18 18:16:17.605948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605859    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy"
Sep 18 18:16:17.605948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605865    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy"
Sep 18 18:16:17.605948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy"
Sep 18 18:16:17.605948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy"
Sep 18 18:16:17.605948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605936    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy"
Sep 18 18:16:17.606140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.605951    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd9deca-585a-445a-ac23-1b2d59c28fb8" containerName="payload-pre-processing"
Sep 18 18:16:17.610474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.610452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.613608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.613590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-94929d65-serving-cert\""
Sep 18 18:16:17.627077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.627056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"]
Sep 18 18:16:17.681220 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.681181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:17.750093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.750055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4mn\" (UniqueName: \"kubernetes.io/projected/080871bd-fcc2-43cf-b242-35063e918b7f-kube-api-access-lc4mn\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.750093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.750093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-metrics-certs\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.750304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.750132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-maas-api-tls\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.750304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.750218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080871bd-fcc2-43cf-b242-35063e918b7f-openshift-service-ca\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.778157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.778122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:17.850952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.850910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-maas-api-tls\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.851133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.851067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080871bd-fcc2-43cf-b242-35063e918b7f-openshift-service-ca\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.851251 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.851226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4mn\" (UniqueName: \"kubernetes.io/projected/080871bd-fcc2-43cf-b242-35063e918b7f-kube-api-access-lc4mn\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.851321 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.851274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-metrics-certs\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.851734 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.851711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080871bd-fcc2-43cf-b242-35063e918b7f-openshift-service-ca\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.853421 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.853402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-maas-api-tls\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.853532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.853516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-metrics-certs\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.859109 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.859057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4mn\" (UniqueName: \"kubernetes.io/projected/080871bd-fcc2-43cf-b242-35063e918b7f-kube-api-access-lc4mn\") pod \"maas-api-e2e-derive-94929d65-665c46f547-csbqr\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:17.880518 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.880489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:17.910375 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.910350    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" path="/var/lib/kubelet/pods/38163c0f-923c-484f-a54a-191c0ca724a3/volumes"
Sep 18 18:16:17.920400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:17.920385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:18.190480 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.190436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:18.199255 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.199229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"]
Sep 18 18:16:18.202605 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:16:18.202576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080871bd_fcc2_43cf_b242_35063e918b7f.slice/crio-306b3b25c1ce515b1928a7f60d702be55b30c2c8e4cd40fed371eec67fe1f6b7 WatchSource:0}: Error finding container 306b3b25c1ce515b1928a7f60d702be55b30c2c8e4cd40fed371eec67fe1f6b7: Status 404 returned error can't find the container with id 306b3b25c1ce515b1928a7f60d702be55b30c2c8e4cd40fed371eec67fe1f6b7
Sep 18 18:16:18.287325 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.287285    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"]
Sep 18 18:16:18.287645 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.287617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:18.287873 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.287855    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38163c0f-923c-484f-a54a-191c0ca724a3" containerName="istio-proxy"
Sep 18 18:16:18.292064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.292021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.294076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.294057    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-94929d65\""
Sep 18 18:16:18.294151 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.294100    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-94929d65-dockercfg-49jjc\""
Sep 18 18:16:18.300746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.300725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"]
Sep 18 18:16:18.327433 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.327401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"]
Sep 18 18:16:18.332336 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.332310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.337347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.337328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"]
Sep 18 18:16:18.458536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.458454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.458536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.458497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blcln\" (UniqueName: \"kubernetes.io/projected/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-kube-api-access-blcln\") pod \"payload-processing-e2e-derive-94929d65-7585756849-6qvfk\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") " pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.458713 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.458589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-plugins-config-volume\") pod \"payload-processing-e2e-derive-94929d65-7585756849-6qvfk\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") " pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.458713 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.458632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbr8\" (UniqueName: \"kubernetes.io/projected/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-kube-api-access-mgbr8\") pod \"payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.559332 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.559300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blcln\" (UniqueName: \"kubernetes.io/projected/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-kube-api-access-blcln\") pod \"payload-processing-e2e-derive-94929d65-7585756849-6qvfk\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") " pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.559493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.559357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-plugins-config-volume\") pod \"payload-processing-e2e-derive-94929d65-7585756849-6qvfk\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") " pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.559493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.559383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbr8\" (UniqueName: \"kubernetes.io/projected/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-kube-api-access-mgbr8\") pod \"payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.559493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.559436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.560112 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.560088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-plugins-config-volume\") pod \"payload-processing-e2e-derive-94929d65-7585756849-6qvfk\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") " pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.560202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.560098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.569233 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.569210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blcln\" (UniqueName: \"kubernetes.io/projected/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-kube-api-access-blcln\") pod \"payload-processing-e2e-derive-94929d65-7585756849-6qvfk\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") " pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.569318 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.569210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbr8\" (UniqueName: \"kubernetes.io/projected/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-kube-api-access-mgbr8\") pod \"payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.603484 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.603461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:18.645375 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.645338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:18.878259 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.878211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"]
Sep 18 18:16:18.880512 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:16:18.880487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e65b874_d0d4_4c68_bbfd_c0c779e4600e.slice/crio-b88ff1bfa626cdcfbba8c163fa4334b95889339118ed8b6275334d64d9961f53 WatchSource:0}: Error finding container b88ff1bfa626cdcfbba8c163fa4334b95889339118ed8b6275334d64d9961f53: Status 404 returned error can't find the container with id b88ff1bfa626cdcfbba8c163fa4334b95889339118ed8b6275334d64d9961f53
Sep 18 18:16:18.925505 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:18.924372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"]
Sep 18 18:16:18.927079 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:16:18.927051    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef081bee_4c2a_4b9b_a2b4_a5a98df1ad57.slice/crio-e3d4126d1be9e1e8a7090caf92b3e78e63a2b117f9a1f01eacaf08d217da25de WatchSource:0}: Error finding container e3d4126d1be9e1e8a7090caf92b3e78e63a2b117f9a1f01eacaf08d217da25de: Status 404 returned error can't find the container with id e3d4126d1be9e1e8a7090caf92b3e78e63a2b117f9a1f01eacaf08d217da25de
Sep 18 18:16:19.117408 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.117369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl" event={"ID":"6e65b874-d0d4-4c68-bbfd-c0c779e4600e","Type":"ContainerStarted","Data":"824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0"}
Sep 18 18:16:19.117608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.117450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl" event={"ID":"6e65b874-d0d4-4c68-bbfd-c0c779e4600e","Type":"ContainerStarted","Data":"b88ff1bfa626cdcfbba8c163fa4334b95889339118ed8b6275334d64d9961f53"}
Sep 18 18:16:19.117608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.117480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:19.118938 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.118913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk" event={"ID":"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57","Type":"ContainerStarted","Data":"9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4"}
Sep 18 18:16:19.119109 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.118959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk" event={"ID":"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57","Type":"ContainerStarted","Data":"e3d4126d1be9e1e8a7090caf92b3e78e63a2b117f9a1f01eacaf08d217da25de"}
Sep 18 18:16:19.119109 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.119013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:19.120485 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.120464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr" event={"ID":"080871bd-fcc2-43cf-b242-35063e918b7f","Type":"ContainerStarted","Data":"631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4"}
Sep 18 18:16:19.120576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.120504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr" event={"ID":"080871bd-fcc2-43cf-b242-35063e918b7f","Type":"ContainerStarted","Data":"306b3b25c1ce515b1928a7f60d702be55b30c2c8e4cd40fed371eec67fe1f6b7"}
Sep 18 18:16:19.120576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.120545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:19.133337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.133252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl" podStartSLOduration=1.133239529 podStartE2EDuration="1.133239529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:16:19.131405449 +0000 UTC m=+949.283280284" watchObservedRunningTime="2026-09-18 18:16:19.133239529 +0000 UTC m=+949.285114363"
Sep 18 18:16:19.149444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.149400    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr" podStartSLOduration=1.783168176 podStartE2EDuration="2.149386069s" totalImagesPullingTime="366.217893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:16:19.14756256 +0000 UTC m=+949.299437406" watchObservedRunningTime="2026-09-18 18:16:19.149386069 +0000 UTC m=+949.301261233"
Sep 18 18:16:19.164222 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:19.164176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk" podStartSLOduration=1.164162814 podStartE2EDuration="1.164162814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:16:19.161844466 +0000 UTC m=+949.313719303" watchObservedRunningTime="2026-09-18 18:16:19.164162814 +0000 UTC m=+949.316037649"
Sep 18 18:16:25.134536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:25.134509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:30.130698 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:30.130664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:30.131228 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:30.131093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:39.799978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:39.799941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:40.478715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:40.478682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:41.200835 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:41.200797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:41.914978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:41.914941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:42.571318 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:42.571288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:43.417483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:43.417447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:44.075730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:44.075689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:45.948584 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:45.948544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:46.531532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.531495    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"]
Sep 18 18:16:46.531838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.531781    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr" podUID="080871bd-fcc2-43cf-b242-35063e918b7f" containerName="maas-api" containerID="cri-o://631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4" gracePeriod=30
Sep 18 18:16:46.565510 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.565478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:46.593298 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.593267    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"]
Sep 18 18:16:46.593582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.593524    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk" podUID="ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" containerName="payload-processing" containerID="cri-o://9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4" gracePeriod=30
Sep 18 18:16:46.600781 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.600747    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"]
Sep 18 18:16:46.600993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.600966    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl" podUID="6e65b874-d0d4-4c68-bbfd-c0c779e4600e" containerName="payload-pre-processing" containerID="cri-o://824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0" gracePeriod=30
Sep 18 18:16:46.633730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.633676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:46.697267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.697227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:46.793012 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.792940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:46.853314 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.853284    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:46.863348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.863141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:46.882600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.882573    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:46.902632 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.902395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:46.915106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.914995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-maas-api-tls\") pod \"080871bd-fcc2-43cf-b242-35063e918b7f\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") "
Sep 18 18:16:46.915106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.915090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080871bd-fcc2-43cf-b242-35063e918b7f-openshift-service-ca\") pod \"080871bd-fcc2-43cf-b242-35063e918b7f\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") "
Sep 18 18:16:46.915374 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.915131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-metrics-certs\") pod \"080871bd-fcc2-43cf-b242-35063e918b7f\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") "
Sep 18 18:16:46.915374 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.915188    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc4mn\" (UniqueName: \"kubernetes.io/projected/080871bd-fcc2-43cf-b242-35063e918b7f-kube-api-access-lc4mn\") pod \"080871bd-fcc2-43cf-b242-35063e918b7f\" (UID: \"080871bd-fcc2-43cf-b242-35063e918b7f\") "
Sep 18 18:16:46.916617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.916207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080871bd-fcc2-43cf-b242-35063e918b7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "080871bd-fcc2-43cf-b242-35063e918b7f" (UID: "080871bd-fcc2-43cf-b242-35063e918b7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:16:46.918410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.918365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "080871bd-fcc2-43cf-b242-35063e918b7f" (UID: "080871bd-fcc2-43cf-b242-35063e918b7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:16:46.918806 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.918766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080871bd-fcc2-43cf-b242-35063e918b7f-kube-api-access-lc4mn" (OuterVolumeSpecName: "kube-api-access-lc4mn") pod "080871bd-fcc2-43cf-b242-35063e918b7f" (UID: "080871bd-fcc2-43cf-b242-35063e918b7f"). InnerVolumeSpecName "kube-api-access-lc4mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:16:46.919870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.919847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "080871bd-fcc2-43cf-b242-35063e918b7f" (UID: "080871bd-fcc2-43cf-b242-35063e918b7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:16:46.946159 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:46.946130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:47.016273 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blcln\" (UniqueName: \"kubernetes.io/projected/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-kube-api-access-blcln\") pod \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") "
Sep 18 18:16:47.016658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgbr8\" (UniqueName: \"kubernetes.io/projected/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-kube-api-access-mgbr8\") pod \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") "
Sep 18 18:16:47.016658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-plugins-config-volume\") pod \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\" (UID: \"6e65b874-d0d4-4c68-bbfd-c0c779e4600e\") "
Sep 18 18:16:47.016658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-plugins-config-volume\") pod \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\" (UID: \"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57\") "
Sep 18 18:16:47.016658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016623    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.016658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016642    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080871bd-fcc2-43cf-b242-35063e918b7f-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.016658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016656    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080871bd-fcc2-43cf-b242-35063e918b7f-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.016887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016669    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4mn\" (UniqueName: \"kubernetes.io/projected/080871bd-fcc2-43cf-b242-35063e918b7f-kube-api-access-lc4mn\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.016887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e65b874-d0d4-4c68-bbfd-c0c779e4600e" (UID: "6e65b874-d0d4-4c68-bbfd-c0c779e4600e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:16:47.016887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.016810    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" (UID: "ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:16:47.018235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.018213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-kube-api-access-blcln" (OuterVolumeSpecName: "kube-api-access-blcln") pod "ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" (UID: "ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57"). InnerVolumeSpecName "kube-api-access-blcln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:16:47.018582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.018562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-kube-api-access-mgbr8" (OuterVolumeSpecName: "kube-api-access-mgbr8") pod "6e65b874-d0d4-4c68-bbfd-c0c779e4600e" (UID: "6e65b874-d0d4-4c68-bbfd-c0c779e4600e"). InnerVolumeSpecName "kube-api-access-mgbr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:16:47.117917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.117835    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.117917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.117863    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.117917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.117876    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blcln\" (UniqueName: \"kubernetes.io/projected/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57-kube-api-access-blcln\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.117917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.117887    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbr8\" (UniqueName: \"kubernetes.io/projected/6e65b874-d0d4-4c68-bbfd-c0c779e4600e-kube-api-access-mgbr8\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:16:47.228787 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.228751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:47.264210 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.264183    2580 generic.go:356] "Generic (PLEG): container finished" podID="ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" containerID="9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4" exitCode=0
Sep 18 18:16:47.264333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.264227    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"
Sep 18 18:16:47.264333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.264276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk" event={"ID":"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57","Type":"ContainerDied","Data":"9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4"}
Sep 18 18:16:47.264434 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.264336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk" event={"ID":"ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57","Type":"ContainerDied","Data":"e3d4126d1be9e1e8a7090caf92b3e78e63a2b117f9a1f01eacaf08d217da25de"}
Sep 18 18:16:47.264434 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.264361    2580 scope.go:117] "RemoveContainer" containerID="9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4"
Sep 18 18:16:47.265767 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.265738    2580 generic.go:356] "Generic (PLEG): container finished" podID="080871bd-fcc2-43cf-b242-35063e918b7f" containerID="631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4" exitCode=0
Sep 18 18:16:47.265917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.265789    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"
Sep 18 18:16:47.265917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.265783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr" event={"ID":"080871bd-fcc2-43cf-b242-35063e918b7f","Type":"ContainerDied","Data":"631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4"}
Sep 18 18:16:47.266235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.265941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr" event={"ID":"080871bd-fcc2-43cf-b242-35063e918b7f","Type":"ContainerDied","Data":"306b3b25c1ce515b1928a7f60d702be55b30c2c8e4cd40fed371eec67fe1f6b7"}
Sep 18 18:16:47.267470 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.267445    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e65b874-d0d4-4c68-bbfd-c0c779e4600e" containerID="824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0" exitCode=0
Sep 18 18:16:47.267615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.267482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl" event={"ID":"6e65b874-d0d4-4c68-bbfd-c0c779e4600e","Type":"ContainerDied","Data":"824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0"}
Sep 18 18:16:47.267615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.267530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl" event={"ID":"6e65b874-d0d4-4c68-bbfd-c0c779e4600e","Type":"ContainerDied","Data":"b88ff1bfa626cdcfbba8c163fa4334b95889339118ed8b6275334d64d9961f53"}
Sep 18 18:16:47.267615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.267604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"
Sep 18 18:16:47.274365 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.274325    2580 scope.go:117] "RemoveContainer" containerID="9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4"
Sep 18 18:16:47.274819 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:16:47.274767    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4\": container with ID starting with 9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4 not found: ID does not exist" containerID="9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4"
Sep 18 18:16:47.274908 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.274882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4"} err="failed to get container status \"9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4\": rpc error: code = NotFound desc = could not find container \"9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4\": container with ID starting with 9fbecf713f5575243ac94a54ea7b2c195adb18013127beca7e61e24e59aaa9c4 not found: ID does not exist"
Sep 18 18:16:47.274982 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.274909    2580 scope.go:117] "RemoveContainer" containerID="631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4"
Sep 18 18:16:47.283667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.283650    2580 scope.go:117] "RemoveContainer" containerID="631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4"
Sep 18 18:16:47.283914 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:16:47.283897    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4\": container with ID starting with 631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4 not found: ID does not exist" containerID="631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4"
Sep 18 18:16:47.283962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.283922    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4"} err="failed to get container status \"631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4\": rpc error: code = NotFound desc = could not find container \"631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4\": container with ID starting with 631ad6fd46ec14acbd1a389b1c7c230a10c1da1305dc76646df09b244326a1a4 not found: ID does not exist"
Sep 18 18:16:47.283962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.283938    2580 scope.go:117] "RemoveContainer" containerID="824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0"
Sep 18 18:16:47.288514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.288493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"]
Sep 18 18:16:47.292226 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.292205    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-94929d65-7585756849-6qvfk"]
Sep 18 18:16:47.292769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.292565    2580 scope.go:117] "RemoveContainer" containerID="824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0"
Sep 18 18:16:47.292952 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:16:47.292930    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0\": container with ID starting with 824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0 not found: ID does not exist" containerID="824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0"
Sep 18 18:16:47.293016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.292963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0"} err="failed to get container status \"824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0\": rpc error: code = NotFound desc = could not find container \"824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0\": container with ID starting with 824658d4276177ce90b0170d6228003707d6dbdb6ec83274227654ac0dd99cc0 not found: ID does not exist"
Sep 18 18:16:47.305963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.305943    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"]
Sep 18 18:16:47.311459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.311437    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-94929d65-665c46f547-csbqr"]
Sep 18 18:16:47.319615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.319548    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"]
Sep 18 18:16:47.321257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.321237    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-94929d65-7b564c8878-jlrhl"]
Sep 18 18:16:47.634189 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.634148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:47.734179 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.734146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:47.837299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.837223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:47.910880 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.910852    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080871bd-fcc2-43cf-b242-35063e918b7f" path="/var/lib/kubelet/pods/080871bd-fcc2-43cf-b242-35063e918b7f/volumes"
Sep 18 18:16:47.911271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.911258    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e65b874-d0d4-4c68-bbfd-c0c779e4600e" path="/var/lib/kubelet/pods/6e65b874-d0d4-4c68-bbfd-c0c779e4600e/volumes"
Sep 18 18:16:47.911575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:47.911563    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" path="/var/lib/kubelet/pods/ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57/volumes"
Sep 18 18:16:48.137063 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:48.136961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:48.847914 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:48.847871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:49.303353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:49.303314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:49.531320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:49.531284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:50.241708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:50.241670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:53.808907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:53.808874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:55.240894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:55.240859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:55.304616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:55.304581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:55.379886 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:55.379846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:55.480744 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:55.480709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:55.597280 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:55.597195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:55.954532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:55.954494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:58.799907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:58.799857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:58.856563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:58.856532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:59.784852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.784820    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"]
Sep 18 18:16:59.785239 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785225    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e65b874-d0d4-4c68-bbfd-c0c779e4600e" containerName="payload-pre-processing"
Sep 18 18:16:59.785292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785241    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e65b874-d0d4-4c68-bbfd-c0c779e4600e" containerName="payload-pre-processing"
Sep 18 18:16:59.785292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785267    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080871bd-fcc2-43cf-b242-35063e918b7f" containerName="maas-api"
Sep 18 18:16:59.785292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785277    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="080871bd-fcc2-43cf-b242-35063e918b7f" containerName="maas-api"
Sep 18 18:16:59.785292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" containerName="payload-processing"
Sep 18 18:16:59.785292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785292    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" containerName="payload-processing"
Sep 18 18:16:59.785451 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785349    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="080871bd-fcc2-43cf-b242-35063e918b7f" containerName="maas-api"
Sep 18 18:16:59.785451 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef081bee-4c2a-4b9b-a2b4-a5a98df1ad57" containerName="payload-processing"
Sep 18 18:16:59.785451 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.785368    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e65b874-d0d4-4c68-bbfd-c0c779e4600e" containerName="payload-pre-processing"
Sep 18 18:16:59.790004 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.789979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:16:59.792339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.792319    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-039dbc\"/\"kube-root-ca.crt\""
Sep 18 18:16:59.792923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.792904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-039dbc\"/\"default-dockercfg-q6jw7\""
Sep 18 18:16:59.793055 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.792937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-039dbc\"/\"premium-w3-039dbc-kserve-self-signed-certs\""
Sep 18 18:16:59.793055 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.792946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-039dbc\"/\"openshift-service-ca.crt\""
Sep 18 18:16:59.798019 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.797993    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"]
Sep 18 18:16:59.813441 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.813417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:59.932427 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.932395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tmp-dir\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:16:59.932427 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.932431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-home\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:16:59.932621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.932462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tls-certs\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:16:59.932621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.932523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-dshm\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:16:59.932621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.932601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-model-cache\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:16:59.932725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:16:59.932620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcm8\" (UniqueName: \"kubernetes.io/projected/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-kube-api-access-xzcm8\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcm8\" (UniqueName: \"kubernetes.io/projected/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-kube-api-access-xzcm8\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tmp-dir\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-home\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tls-certs\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033498 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-dshm\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-model-cache\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tmp-dir\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-home\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.033975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.033956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-model-cache\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.036145 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.036095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-dshm\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.036359 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.036345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tls-certs\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.041170 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.041148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcm8\" (UniqueName: \"kubernetes.io/projected/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-kube-api-access-xzcm8\") pod \"premium-w3-039dbc-kserve-77875d57b8-rlj54\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") " pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.100639 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.100608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:00.374462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:00.373822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"]
Sep 18 18:17:01.340742 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:01.340706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54" event={"ID":"90b2ebbb-8ddc-4e24-ac16-79655fefdecb","Type":"ContainerStarted","Data":"af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4"}
Sep 18 18:17:01.341201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:01.340755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54" event={"ID":"90b2ebbb-8ddc-4e24-ac16-79655fefdecb","Type":"ContainerStarted","Data":"7faeb316928780cd4d89e892fd2d5ac6ae572306bb1a7937dd8c2a57a28a19dd"}
Sep 18 18:17:01.359753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:01.359697    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54" podStartSLOduration=2.359681909 podStartE2EDuration="2.359681909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:01.357438221 +0000 UTC m=+991.509313056" watchObservedRunningTime="2026-09-18 18:17:01.359681909 +0000 UTC m=+991.511556743"
Sep 18 18:17:10.101118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:10.101088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:10.101523 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:10.101162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:10.113488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:10.113468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:10.404456 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:10.404377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:17:16.073567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.073536    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"]
Sep 18 18:17:16.076966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.076943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.079689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.079664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bcf02f\"/\"default-dockercfg-n5qps\""
Sep 18 18:17:16.079839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.079666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bcf02f\"/\"test-model-bcf02f-kserve-self-signed-certs\""
Sep 18 18:17:16.079839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.079709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bcf02f\"/\"kube-root-ca.crt\""
Sep 18 18:17:16.079839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.079749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bcf02f\"/\"openshift-service-ca.crt\""
Sep 18 18:17:16.086759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.086740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"]
Sep 18 18:17:16.103921 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.103897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:16.168330 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.168296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjc2v\" (UniqueName: \"kubernetes.io/projected/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-kube-api-access-zjc2v\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.168436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.168341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tls-certs\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.168436 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.168360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-home\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.168522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.168438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-model-cache\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.168522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.168496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-dshm\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.168589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.168522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tmp-dir\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.269978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.269948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tls-certs\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.269978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.269984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-home\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270228 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-model-cache\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270228 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-dshm\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270228 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tmp-dir\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270228 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjc2v\" (UniqueName: \"kubernetes.io/projected/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-kube-api-access-zjc2v\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-home\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270523 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-model-cache\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.270568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.270554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tmp-dir\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.272488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.272461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-dshm\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.272902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.272880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tls-certs\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.278311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.278285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjc2v\" (UniqueName: \"kubernetes.io/projected/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-kube-api-access-zjc2v\") pod \"test-model-bcf02f-kserve-5c647c56c-2w69w\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") " pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.387610 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.387548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:16.666883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:16.661057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"]
Sep 18 18:17:16.668502 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:17:16.668472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77b8dc5a_9fa7_41a5_af28_4d9647ebfa8e.slice/crio-d0e32af01b96ca3a7577eef82f1b8adcc6c104171aabf9311a977b4a9ea13f91 WatchSource:0}: Error finding container d0e32af01b96ca3a7577eef82f1b8adcc6c104171aabf9311a977b4a9ea13f91: Status 404 returned error can't find the container with id d0e32af01b96ca3a7577eef82f1b8adcc6c104171aabf9311a977b4a9ea13f91
Sep 18 18:17:17.429821 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:17.429778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w" event={"ID":"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e","Type":"ContainerStarted","Data":"a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1"}
Sep 18 18:17:17.430340 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:17.429832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w" event={"ID":"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e","Type":"ContainerStarted","Data":"d0e32af01b96ca3a7577eef82f1b8adcc6c104171aabf9311a977b4a9ea13f91"}
Sep 18 18:17:17.449213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:17.449161    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w" podStartSLOduration=1.449144185 podStartE2EDuration="1.449144185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:17.447177252 +0000 UTC m=+1007.599052088" watchObservedRunningTime="2026-09-18 18:17:17.449144185 +0000 UTC m=+1007.601019020"
Sep 18 18:17:18.880240 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:18.880205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:21.888213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:21.888175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:26.388669 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:26.388629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:26.389258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:26.388712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:26.402933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:26.402911    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:26.490478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:26.490451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:17:27.648890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:27.648852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:27.759832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:27.759801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:27.848541 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:27.848503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:43.679543 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:43.679489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:49.487868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:49.487832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:49.597545 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:49.597490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:49.689560 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:17:49.689530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:00.159933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:00.159886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:00.914338 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:00.914298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:01.015156 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:01.015122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:02.915522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:02.915483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:07.050353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:07.050317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:07.191021 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:07.190986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:07.291485 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:07.291448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:07.996810 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:07.996767    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:08.089437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:08.089394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:08.193796 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:08.193756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:09.633752 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:09.633706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:15.581609 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:15.581498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:15.643899 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:15.643869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:16.239790 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:16.239748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:22.331535 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.331481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:22.430829 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.430794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:22.551525 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.551487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:22.833919 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.833885    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"]
Sep 18 18:18:22.837534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.837511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.839778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.839756    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-026961cf-openshift-default-dockercfg-z959t\""
Sep 18 18:18:22.851057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.851009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"]
Sep 18 18:18:22.859960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.859924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:22.974136 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-certs\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istiod-ca-cert\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-envoy\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-socket\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-podinfo\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-credential-socket\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974464 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-token\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974464 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-data\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:22.974464 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:22.974400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9tm\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-kube-api-access-gl9tm\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.074988 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.074956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istiod-ca-cert\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.074992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-envoy\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-socket\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075266 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-podinfo\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075266 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-credential-socket\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075266 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-token\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075414 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-data\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075414 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9tm\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-kube-api-access-gl9tm\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-certs\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-socket\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075661 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-credential-socket\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istiod-ca-cert\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.075852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-certs\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.076118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.075959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-data\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.077491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.077458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-envoy\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.077810 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.077789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-podinfo\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.082933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.082909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9tm\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-kube-api-access-gl9tm\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.083384 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.083368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-token\") pod \"e2e-mt-026961cf-openshift-default-845f5469cb-d5968\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") " pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.100180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.100108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:23.149643 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.149619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:23.422945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.422918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"]
Sep 18 18:18:23.426326 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:18:23.426296    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1f3e32_c72a_45b1_9abf_a3837e8a9c19.slice/crio-e28b929d1f21c20a776b5a5d2f532ce67d48e08b5f8a25e94acc20cb2d675382 WatchSource:0}: Error finding container e28b929d1f21c20a776b5a5d2f532ce67d48e08b5f8a25e94acc20cb2d675382: Status 404 returned error can't find the container with id e28b929d1f21c20a776b5a5d2f532ce67d48e08b5f8a25e94acc20cb2d675382
Sep 18 18:18:23.429202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.429159    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:18:23.429285 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.429250    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:18:23.429331 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.429294    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:18:23.770548 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.770464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" event={"ID":"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19","Type":"ContainerStarted","Data":"99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"}
Sep 18 18:18:23.770548 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.770513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" event={"ID":"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19","Type":"ContainerStarted","Data":"e28b929d1f21c20a776b5a5d2f532ce67d48e08b5f8a25e94acc20cb2d675382"}
Sep 18 18:18:23.789349 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:23.789282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podStartSLOduration=1.789263294 podStartE2EDuration="1.789263294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:18:23.788620102 +0000 UTC m=+1073.940494952" watchObservedRunningTime="2026-09-18 18:18:23.789263294 +0000 UTC m=+1073.941138129"
Sep 18 18:18:24.149962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:24.149881    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:24.154757 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:24.154728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:24.775836 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:24.775806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:24.776775 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:24.776759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:25.414238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:25.414200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:25.742887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:25.742786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:25.850784 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:25.850753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:26.421236 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:26.421195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:27.207714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:27.207685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:28.118457 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:28.118415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:28.829449 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:28.829403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:28.927330 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:28.927256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:29.027296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.027264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:29.304868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.304832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:29.391652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.391615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:29.405330 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.405305    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"]
Sep 18 18:18:29.408935 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.408920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.411048 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.411007    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-026961cf-dockercfg-gd7p9\""
Sep 18 18:18:29.411223 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.411203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-026961cf\""
Sep 18 18:18:29.419064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.419016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"]
Sep 18 18:18:29.547299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.547264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6l49\" (UniqueName: \"kubernetes.io/projected/e4ab06d2-4c78-4f74-81cf-85644aba90ce-kube-api-access-g6l49\") pod \"payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.547457 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.547319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ab06d2-4c78-4f74-81cf-85644aba90ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.648311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.648227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6l49\" (UniqueName: \"kubernetes.io/projected/e4ab06d2-4c78-4f74-81cf-85644aba90ce-kube-api-access-g6l49\") pod \"payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.648311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.648293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ab06d2-4c78-4f74-81cf-85644aba90ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.648964 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.648945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ab06d2-4c78-4f74-81cf-85644aba90ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.656051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.655994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6l49\" (UniqueName: \"kubernetes.io/projected/e4ab06d2-4c78-4f74-81cf-85644aba90ce-kube-api-access-g6l49\") pod \"payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.720848 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.720820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:29.996696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:29.996674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"]
Sep 18 18:18:29.998969 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:18:29.998937    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ab06d2_4c78_4f74_81cf_85644aba90ce.slice/crio-6f2209cab6fe8d3a39a713050fd1f9317339f37131c886d305ca1a6029371786 WatchSource:0}: Error finding container 6f2209cab6fe8d3a39a713050fd1f9317339f37131c886d305ca1a6029371786: Status 404 returned error can't find the container with id 6f2209cab6fe8d3a39a713050fd1f9317339f37131c886d305ca1a6029371786
Sep 18 18:18:30.810940 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:30.810901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795" event={"ID":"e4ab06d2-4c78-4f74-81cf-85644aba90ce","Type":"ContainerStarted","Data":"9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a"}
Sep 18 18:18:30.811160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:30.810955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795" event={"ID":"e4ab06d2-4c78-4f74-81cf-85644aba90ce","Type":"ContainerStarted","Data":"6f2209cab6fe8d3a39a713050fd1f9317339f37131c886d305ca1a6029371786"}
Sep 18 18:18:30.811160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:30.811000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:36.382110 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:36.382078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:41.818577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:41.818547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:18:41.837428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:41.837366    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795" podStartSLOduration=12.837346254 podStartE2EDuration="12.837346254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:18:30.827832033 +0000 UTC m=+1080.979706869" watchObservedRunningTime="2026-09-18 18:18:41.837346254 +0000 UTC m=+1091.989221090"
Sep 18 18:18:48.914761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:48.914730    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerID="99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea" exitCode=0
Sep 18 18:18:48.915350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:48.914803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" event={"ID":"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19","Type":"ContainerDied","Data":"99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"}
Sep 18 18:18:48.915350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:48.915127    2580 scope.go:117] "RemoveContainer" containerID="99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"
Sep 18 18:18:48.916249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:48.916218    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:18:48.916333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:48.916290    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:18:48.916333 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:48.916324    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:18:49.921879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:49.921841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" event={"ID":"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19","Type":"ContainerStarted","Data":"431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e"}
Sep 18 18:18:50.151250 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:50.151192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:50.151534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:50.151501    2580 patch_prober.go:28] interesting pod/e2e-mt-026961cf-openshift-default-845f5469cb-d5968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 18 18:18:50.151609 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:50.151571    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 18 18:18:50.668089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:50.668043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:50.768720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:50.768680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.150691 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:51.150590    2580 patch_prober.go:28] interesting pod/e2e-mt-026961cf-openshift-default-845f5469cb-d5968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 18 18:18:51.150691 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:51.150663    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 18 18:18:51.312047 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:51.311995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.411776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:51.411741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.512618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:51.512590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.736116 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:51.735996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:52.150797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:52.150692    2580 patch_prober.go:28] interesting pod/e2e-mt-026961cf-openshift-default-845f5469cb-d5968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 18 18:18:52.151326 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:52.150787    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 18 18:18:52.319086 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:52.319054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:53.061067 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:53.061012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:53.150083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:53.150014    2580 patch_prober.go:28] interesting pod/e2e-mt-026961cf-openshift-default-845f5469cb-d5968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 18 18:18:53.150420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:53.150386    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 18 18:18:53.814641 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:53.814602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:54.150208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:54.150089    2580 patch_prober.go:28] interesting pod/e2e-mt-026961cf-openshift-default-845f5469cb-d5968 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 18 18:18:54.150500 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:54.150450    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 18 18:18:54.464939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:54.464907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:54.776471 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:54.776398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:55.339638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:55.339606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:55.342099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:55.342068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:18:57.553750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:57.553712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:57.654784 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:57.654749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:58.577391 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.577358    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"]
Sep 18 18:18:58.581191 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.581169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.583579 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.583559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-37c153\"/\"kube-root-ca.crt\""
Sep 18 18:18:58.584241 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.584218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-37c153\"/\"distinct-w0-37c153-kserve-self-signed-certs\""
Sep 18 18:18:58.584351 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.584260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-37c153\"/\"openshift-service-ca.crt\""
Sep 18 18:18:58.584420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.584347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-37c153\"/\"default-dockercfg-82vrd\""
Sep 18 18:18:58.589745 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.589727    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"]
Sep 18 18:18:58.606709 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.606681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:58.623933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.623908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-home\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.624104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.623952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-model-cache\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.624104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.623999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgs9\" (UniqueName: \"kubernetes.io/projected/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-kube-api-access-gsgs9\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.624104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.624070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-dshm\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.624299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.624163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tls-certs\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.624299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.624192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tmp-dir\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.681117 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.681080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:58.725139 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tmp-dir\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-home\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-model-cache\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgs9\" (UniqueName: \"kubernetes.io/projected/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-kube-api-access-gsgs9\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-dshm\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tls-certs\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-home\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725735 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tmp-dir\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.725939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.725894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-model-cache\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.728571 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.728546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-dshm\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.728812 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.728794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tls-certs\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.738205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.738171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgs9\" (UniqueName: \"kubernetes.io/projected/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-kube-api-access-gsgs9\") pod \"distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") " pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:58.891753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:58.891670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:18:59.164471 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:59.164446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"]
Sep 18 18:18:59.166396 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:18:59.166358    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3513d1c4_6eb3_4013_a53a_a5408b1a2e62.slice/crio-29bf0be4c44a438319e435fd0da85e3ab31c2da5d8a19e89885b26d0b1d4ef0d WatchSource:0}: Error finding container 29bf0be4c44a438319e435fd0da85e3ab31c2da5d8a19e89885b26d0b1d4ef0d: Status 404 returned error can't find the container with id 29bf0be4c44a438319e435fd0da85e3ab31c2da5d8a19e89885b26d0b1d4ef0d
Sep 18 18:18:59.433162 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:59.433061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:59.985172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:59.985137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz" event={"ID":"3513d1c4-6eb3-4013-a53a-a5408b1a2e62","Type":"ContainerStarted","Data":"0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34"}
Sep 18 18:18:59.985172 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:18:59.985176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz" event={"ID":"3513d1c4-6eb3-4013-a53a-a5408b1a2e62","Type":"ContainerStarted","Data":"29bf0be4c44a438319e435fd0da85e3ab31c2da5d8a19e89885b26d0b1d4ef0d"}
Sep 18 18:19:00.001244 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:00.001194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz" podStartSLOduration=2.001179496 podStartE2EDuration="2.001179496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:00.000078803 +0000 UTC m=+1110.151953664" watchObservedRunningTime="2026-09-18 18:19:00.001179496 +0000 UTC m=+1110.153054331"
Sep 18 18:19:00.048324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:00.048284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:00.891868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:00.891820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:01.581182 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:01.581141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:02.233276 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:02.233236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:08.543459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.543419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:08.576970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.576772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"]
Sep 18 18:19:08.577177 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.577107    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795" podUID="e4ab06d2-4c78-4f74-81cf-85644aba90ce" containerName="payload-pre-processing" containerID="cri-o://9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a" gracePeriod=30
Sep 18 18:19:08.607658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.607600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:08.707652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.707617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:08.765615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.765583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:08.827826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.827803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:19:08.892300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.892266    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:19:08.892300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.892305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:19:08.905991 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.905959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:19:08.928559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.928527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ab06d2-4c78-4f74-81cf-85644aba90ce-plugins-config-volume\") pod \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") "
Sep 18 18:19:08.928739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.928580    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6l49\" (UniqueName: \"kubernetes.io/projected/e4ab06d2-4c78-4f74-81cf-85644aba90ce-kube-api-access-g6l49\") pod \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\" (UID: \"e4ab06d2-4c78-4f74-81cf-85644aba90ce\") "
Sep 18 18:19:08.929375 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.929348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ab06d2-4c78-4f74-81cf-85644aba90ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4ab06d2-4c78-4f74-81cf-85644aba90ce" (UID: "e4ab06d2-4c78-4f74-81cf-85644aba90ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:19:08.931075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:08.931045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ab06d2-4c78-4f74-81cf-85644aba90ce-kube-api-access-g6l49" (OuterVolumeSpecName: "kube-api-access-g6l49") pod "e4ab06d2-4c78-4f74-81cf-85644aba90ce" (UID: "e4ab06d2-4c78-4f74-81cf-85644aba90ce"). InnerVolumeSpecName "kube-api-access-g6l49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:09.030016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.029760    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ab06d2-4c78-4f74-81cf-85644aba90ce-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:09.030016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.029790    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6l49\" (UniqueName: \"kubernetes.io/projected/e4ab06d2-4c78-4f74-81cf-85644aba90ce-kube-api-access-g6l49\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:09.034887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.034858    2580 generic.go:356] "Generic (PLEG): container finished" podID="e4ab06d2-4c78-4f74-81cf-85644aba90ce" containerID="9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a" exitCode=0
Sep 18 18:19:09.035051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.034915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"
Sep 18 18:19:09.035051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.034938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795" event={"ID":"e4ab06d2-4c78-4f74-81cf-85644aba90ce","Type":"ContainerDied","Data":"9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a"}
Sep 18 18:19:09.035051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.034988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795" event={"ID":"e4ab06d2-4c78-4f74-81cf-85644aba90ce","Type":"ContainerDied","Data":"6f2209cab6fe8d3a39a713050fd1f9317339f37131c886d305ca1a6029371786"}
Sep 18 18:19:09.035051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.035010    2580 scope.go:117] "RemoveContainer" containerID="9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a"
Sep 18 18:19:09.044588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.044570    2580 scope.go:117] "RemoveContainer" containerID="9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a"
Sep 18 18:19:09.044892 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:19:09.044871    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a\": container with ID starting with 9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a not found: ID does not exist" containerID="9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a"
Sep 18 18:19:09.044988 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.044900    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a"} err="failed to get container status \"9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a\": rpc error: code = NotFound desc = could not find container \"9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a\": container with ID starting with 9e7379f83bbff160763ea8434614e3ab382a718b22e6c28d76cf826f0f584e4a not found: ID does not exist"
Sep 18 18:19:09.047985 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.047963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:19:09.058177 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.058126    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"]
Sep 18 18:19:09.063490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.063469    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-026961cf-fd9d7f66b-b8795"]
Sep 18 18:19:09.130800 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.130746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.196496 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.196454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.910921 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:09.910889    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ab06d2-4c78-4f74-81cf-85644aba90ce" path="/var/lib/kubelet/pods/e4ab06d2-4c78-4f74-81cf-85644aba90ce/volumes"
Sep 18 18:19:10.119563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:10.119531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:10.914590 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:10.914547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:11.681580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:11.681541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:12.196426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:12.196382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:14.907674 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:14.907627    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:14.984783 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:14.984751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:15.142518 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:15.142480    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"]
Sep 18 18:19:15.142754 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:15.142730    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy" containerID="cri-o://431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e" gracePeriod=30
Sep 18 18:19:15.157771 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:15.157685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:15.503292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:15.503201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:15.599238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:15.599193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:15.707458 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:15.707423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:20.395761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.395734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:19:20.431557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.431522    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"]
Sep 18 18:19:20.442990 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.442951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:20.540119 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540078    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-credential-socket\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540119 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-token\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-envoy\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-data\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540198    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istiod-ca-cert\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-certs\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:20.540616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-podinfo\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:20.540616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl9tm\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-kube-api-access-gl9tm\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-socket\") pod \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\" (UID: \"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19\") "
Sep 18 18:19:20.540616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:20.540844 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:19:20.540844 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.540824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:20.541125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.541106    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.541267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.541129    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.541267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.541144    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.541267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.541157    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.541267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.541169    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.542611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.542590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:20.542759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.542736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-kube-api-access-gl9tm" (OuterVolumeSpecName: "kube-api-access-gl9tm") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "kube-api-access-gl9tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:20.543052 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.543013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:19:20.543052 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.543020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" (UID: "2c1f3e32-c72a-45b1-9abf-a3837e8a9c19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:20.642070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.642041    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.642070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.642066    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.642070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.642074    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:20.642268 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:20.642082    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl9tm\" (UniqueName: \"kubernetes.io/projected/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19-kube-api-access-gl9tm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.097871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.097834    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerID="431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e" exitCode=0
Sep 18 18:19:21.098092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.097886    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"
Sep 18 18:19:21.098092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.097922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" event={"ID":"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19","Type":"ContainerDied","Data":"431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e"}
Sep 18 18:19:21.098092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.097981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968" event={"ID":"2c1f3e32-c72a-45b1-9abf-a3837e8a9c19","Type":"ContainerDied","Data":"e28b929d1f21c20a776b5a5d2f532ce67d48e08b5f8a25e94acc20cb2d675382"}
Sep 18 18:19:21.098092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.098002    2580 scope.go:117] "RemoveContainer" containerID="431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e"
Sep 18 18:19:21.107293 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.107275    2580 scope.go:117] "RemoveContainer" containerID="99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"
Sep 18 18:19:21.115581 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.115564    2580 scope.go:117] "RemoveContainer" containerID="431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e"
Sep 18 18:19:21.115858 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:19:21.115839    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e\": container with ID starting with 431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e not found: ID does not exist" containerID="431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e"
Sep 18 18:19:21.115922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.115867    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e"} err="failed to get container status \"431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e\": rpc error: code = NotFound desc = could not find container \"431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e\": container with ID starting with 431786abf747125a95da63405e15b4a82c2770198e42d118f359723a6883af6e not found: ID does not exist"
Sep 18 18:19:21.115922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.115884    2580 scope.go:117] "RemoveContainer" containerID="99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"
Sep 18 18:19:21.116237 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:19:21.116189    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea\": container with ID starting with 99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea not found: ID does not exist" containerID="99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"
Sep 18 18:19:21.116276 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.116246    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea"} err="failed to get container status \"99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea\": rpc error: code = NotFound desc = could not find container \"99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea\": container with ID starting with 99adb9603e644d562f9604c124e47ad216e0d73b97505e7e2591c6902a0e79ea not found: ID does not exist"
Sep 18 18:19:21.118830 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.118809    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"]
Sep 18 18:19:21.123420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.123397    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-026961cf-openshift-default-845f5469cb-d5968"]
Sep 18 18:19:21.600958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.600923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:21.715808 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.715763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:21.808901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.808868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:21.910843 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.910812    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" path="/var/lib/kubelet/pods/2c1f3e32-c72a-45b1-9abf-a3837e8a9c19/volumes"
Sep 18 18:19:21.938215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:21.938175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:27.958746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:27.958704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.056461 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:28.056423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.857405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:28.857368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:29.130181 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130099    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"]
Sep 18 18:19:29.130708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130646    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ab06d2-4c78-4f74-81cf-85644aba90ce" containerName="payload-pre-processing"
Sep 18 18:19:29.130708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130669    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ab06d2-4c78-4f74-81cf-85644aba90ce" containerName="payload-pre-processing"
Sep 18 18:19:29.130708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130689    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy"
Sep 18 18:19:29.130708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130697    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy"
Sep 18 18:19:29.130708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy"
Sep 18 18:19:29.131077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130717    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy"
Sep 18 18:19:29.131077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130849    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy"
Sep 18 18:19:29.131077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130860    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ab06d2-4c78-4f74-81cf-85644aba90ce" containerName="payload-pre-processing"
Sep 18 18:19:29.131077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.130873    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1f3e32-c72a-45b1-9abf-a3837e8a9c19" containerName="istio-proxy"
Sep 18 18:19:29.135903 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.135878    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.138659 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.138420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f5e6d4-openshift-default-dockercfg-555cc\""
Sep 18 18:19:29.145377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.145355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"]
Sep 18 18:19:29.160726 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.160646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:29.222712 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-socket\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.222712 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d3295f-369d-4e08-b165-3b94581287eb-istio-podinfo\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.222888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-envoy\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.222888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-data\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.222888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-istio-token\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.222888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d3295f-369d-4e08-b165-3b94581287eb-istiod-ca-cert\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.223100 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbgp\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-kube-api-access-dqbgp\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.223100 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-credential-socket\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.223100 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.222953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-certs\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324666 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-envoy\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324666 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-data\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-istio-token\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d3295f-369d-4e08-b165-3b94581287eb-istiod-ca-cert\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbgp\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-kube-api-access-dqbgp\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-credential-socket\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.324900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-certs\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.325205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-socket\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.325205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.324980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d3295f-369d-4e08-b165-3b94581287eb-istio-podinfo\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.325205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.325162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-data\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.325666 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.325645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-certs\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.326078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.326043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-credential-socket\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.326381 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.326347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d3295f-369d-4e08-b165-3b94581287eb-istiod-ca-cert\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.326567 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.326549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-socket\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.327494 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.327464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-envoy\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.327931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.327911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d3295f-369d-4e08-b165-3b94581287eb-istio-podinfo\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.332761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.332739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-istio-token\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.333435 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.333415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbgp\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-kube-api-access-dqbgp\") pod \"e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") " pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.448421 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.448373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:29.492449 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.492407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:29.724556 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.724526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"]
Sep 18 18:19:29.726816 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:19:29.726783    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d3295f_369d_4e08_b165_3b94581287eb.slice/crio-924a729dcbb7739b95b5a8148139439349959e330f17cd395a1095a9ec5a7d55 WatchSource:0}: Error finding container 924a729dcbb7739b95b5a8148139439349959e330f17cd395a1095a9ec5a7d55: Status 404 returned error can't find the container with id 924a729dcbb7739b95b5a8148139439349959e330f17cd395a1095a9ec5a7d55
Sep 18 18:19:29.729756 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.729716    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:29.729879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.729812    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:29.729879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:29.729868    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:30.147843 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:30.147751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" event={"ID":"f5d3295f-369d-4e08-b165-3b94581287eb","Type":"ContainerStarted","Data":"733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"}
Sep 18 18:19:30.147843 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:30.147804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" event={"ID":"f5d3295f-369d-4e08-b165-3b94581287eb","Type":"ContainerStarted","Data":"924a729dcbb7739b95b5a8148139439349959e330f17cd395a1095a9ec5a7d55"}
Sep 18 18:19:30.167153 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:30.167074    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podStartSLOduration=1.167053812 podStartE2EDuration="1.167053812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:30.165487598 +0000 UTC m=+1140.317362433" watchObservedRunningTime="2026-09-18 18:19:30.167053812 +0000 UTC m=+1140.318928647"
Sep 18 18:19:30.449572 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:30.449535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:30.454652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:30.454626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:31.152930 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:31.152896    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:31.153799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:31.153782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:31.317695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:31.317657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:32.114365 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:32.114333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:32.236692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:32.236654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:33.014243 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:33.014202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:33.902347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:33.902311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:34.450575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:34.450539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:34.563840 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:34.563804    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"]
Sep 18 18:19:34.588265 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:34.588226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:35.233830 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.233792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:35.334184 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.334147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:35.451710 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.451676    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt" podUID="d2e0c651-dfea-4c75-a64c-f0815b86f287" containerName="main" containerID="cri-o://7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7" gracePeriod=45
Sep 18 18:19:35.529916 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.529801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:35.698357 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.698332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:35.711450 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.711426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:19:35.791811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-home\") pod \"d2e0c651-dfea-4c75-a64c-f0815b86f287\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") "
Sep 18 18:19:35.791811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-tmp-dir\") pod \"d2e0c651-dfea-4c75-a64c-f0815b86f287\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") "
Sep 18 18:19:35.791811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791790    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-model-cache\") pod \"d2e0c651-dfea-4c75-a64c-f0815b86f287\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") "
Sep 18 18:19:35.792089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0c651-dfea-4c75-a64c-f0815b86f287-tls-certs\") pod \"d2e0c651-dfea-4c75-a64c-f0815b86f287\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") "
Sep 18 18:19:35.792089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-dshm\") pod \"d2e0c651-dfea-4c75-a64c-f0815b86f287\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") "
Sep 18 18:19:35.792089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlbgr\" (UniqueName: \"kubernetes.io/projected/d2e0c651-dfea-4c75-a64c-f0815b86f287-kube-api-access-zlbgr\") pod \"d2e0c651-dfea-4c75-a64c-f0815b86f287\" (UID: \"d2e0c651-dfea-4c75-a64c-f0815b86f287\") "
Sep 18 18:19:35.792089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.791999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2e0c651-dfea-4c75-a64c-f0815b86f287" (UID: "d2e0c651-dfea-4c75-a64c-f0815b86f287"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:35.792089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.792067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-home" (OuterVolumeSpecName: "home") pod "d2e0c651-dfea-4c75-a64c-f0815b86f287" (UID: "d2e0c651-dfea-4c75-a64c-f0815b86f287"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:35.792291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.792134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2e0c651-dfea-4c75-a64c-f0815b86f287" (UID: "d2e0c651-dfea-4c75-a64c-f0815b86f287"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:35.792356 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.792339    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:35.792393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.792362    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:35.792393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.792374    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:35.794298 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.794268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-dshm" (OuterVolumeSpecName: "dshm") pod "d2e0c651-dfea-4c75-a64c-f0815b86f287" (UID: "d2e0c651-dfea-4c75-a64c-f0815b86f287"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:35.794413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.794309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e0c651-dfea-4c75-a64c-f0815b86f287-kube-api-access-zlbgr" (OuterVolumeSpecName: "kube-api-access-zlbgr") pod "d2e0c651-dfea-4c75-a64c-f0815b86f287" (UID: "d2e0c651-dfea-4c75-a64c-f0815b86f287"). InnerVolumeSpecName "kube-api-access-zlbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:35.794413 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.794345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e0c651-dfea-4c75-a64c-f0815b86f287-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2e0c651-dfea-4c75-a64c-f0815b86f287" (UID: "d2e0c651-dfea-4c75-a64c-f0815b86f287"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:35.803834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.803806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:35.812206 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.812185    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"]
Sep 18 18:19:35.812580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.812568    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e0c651-dfea-4c75-a64c-f0815b86f287" containerName="main"
Sep 18 18:19:35.812625 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.812583    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e0c651-dfea-4c75-a64c-f0815b86f287" containerName="main"
Sep 18 18:19:35.812665 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.812656    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e0c651-dfea-4c75-a64c-f0815b86f287" containerName="main"
Sep 18 18:19:35.815622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.815609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:35.817717 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.817693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f5e6d4\""
Sep 18 18:19:35.817825 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.817796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f5e6d4-dockercfg-wwjrg\""
Sep 18 18:19:35.825805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.825783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"]
Sep 18 18:19:35.837151 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.837123    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"]
Sep 18 18:19:35.840768 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.840741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:35.852494 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.852470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"]
Sep 18 18:19:35.893648 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:35.893787 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtt6n\" (UniqueName: \"kubernetes.io/projected/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-kube-api-access-qtt6n\") pod \"payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:35.893787 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf9bfe-97b6-466d-89d3-29b7542a8300-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:35.893787 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5xg\" (UniqueName: \"kubernetes.io/projected/04cf9bfe-97b6-466d-89d3-29b7542a8300-kube-api-access-6v5xg\") pod \"payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:35.893907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893788    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbgr\" (UniqueName: \"kubernetes.io/projected/d2e0c651-dfea-4c75-a64c-f0815b86f287-kube-api-access-zlbgr\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:35.893907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893814    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0c651-dfea-4c75-a64c-f0815b86f287-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:35.893907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.893827    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0c651-dfea-4c75-a64c-f0815b86f287-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:35.995156 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.995120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:35.995350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.995206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtt6n\" (UniqueName: \"kubernetes.io/projected/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-kube-api-access-qtt6n\") pod \"payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:35.995350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.995232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf9bfe-97b6-466d-89d3-29b7542a8300-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:35.995350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.995248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5xg\" (UniqueName: \"kubernetes.io/projected/04cf9bfe-97b6-466d-89d3-29b7542a8300-kube-api-access-6v5xg\") pod \"payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:35.995930 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.995906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:35.996072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:35.996053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf9bfe-97b6-466d-89d3-29b7542a8300-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:36.003132 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.003114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtt6n\" (UniqueName: \"kubernetes.io/projected/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-kube-api-access-qtt6n\") pod \"payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:36.003194 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.003178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5xg\" (UniqueName: \"kubernetes.io/projected/04cf9bfe-97b6-466d-89d3-29b7542a8300-kube-api-access-6v5xg\") pod \"payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:36.104672 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.104589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:36.127141 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.127115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:36.152161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.152136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:36.184771 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.184653    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2e0c651-dfea-4c75-a64c-f0815b86f287" containerID="7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7" exitCode=0
Sep 18 18:19:36.184944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.184768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt" event={"ID":"d2e0c651-dfea-4c75-a64c-f0815b86f287","Type":"ContainerDied","Data":"7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7"}
Sep 18 18:19:36.184944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.184795    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"
Sep 18 18:19:36.184944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.184835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt" event={"ID":"d2e0c651-dfea-4c75-a64c-f0815b86f287","Type":"ContainerDied","Data":"25e2a6ef298eac948d9fe140a0c4596845cc5133b763dbd2ca3109d20d389f2c"}
Sep 18 18:19:36.184944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.184861    2580 scope.go:117] "RemoveContainer" containerID="7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7"
Sep 18 18:19:36.198750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.198719    2580 scope.go:117] "RemoveContainer" containerID="7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7"
Sep 18 18:19:36.199265 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:19:36.199227    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7\": container with ID starting with 7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7 not found: ID does not exist" containerID="7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7"
Sep 18 18:19:36.199374 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.199282    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7"} err="failed to get container status \"7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7\": rpc error: code = NotFound desc = could not find container \"7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7\": container with ID starting with 7f05eb698bb5bda9ac962620211217f87ed4197cdc8766402e09387b35b4dcb7 not found: ID does not exist"
Sep 18 18:19:36.203561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.203535    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"]
Sep 18 18:19:36.206115 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.206093    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-de7535/test-model-de7535-kserve-7cb58cbfb7-4cjmt"]
Sep 18 18:19:36.402415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.402386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"]
Sep 18 18:19:36.404517 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:19:36.404495    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cf9bfe_97b6_466d_89d3_29b7542a8300.slice/crio-0d50183edec90892204ba2aa401d36eb6ac37e61bddb30b6cc51051f11c0ffb3 WatchSource:0}: Error finding container 0d50183edec90892204ba2aa401d36eb6ac37e61bddb30b6cc51051f11c0ffb3: Status 404 returned error can't find the container with id 0d50183edec90892204ba2aa401d36eb6ac37e61bddb30b6cc51051f11c0ffb3
Sep 18 18:19:36.444016 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.443996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"]
Sep 18 18:19:36.447144 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:19:36.447115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0d503f_5a10_465c_b0b2_abe2bdc1c54e.slice/crio-f5bf1e72c730c9815df5aadd5b9cd1ecee8883250524578019432c005e69d69e WatchSource:0}: Error finding container f5bf1e72c730c9815df5aadd5b9cd1ecee8883250524578019432c005e69d69e: Status 404 returned error can't find the container with id f5bf1e72c730c9815df5aadd5b9cd1ecee8883250524578019432c005e69d69e
Sep 18 18:19:36.936793 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:36.936757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:37.192721 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.192624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt" event={"ID":"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e","Type":"ContainerStarted","Data":"0101f50a7c71aab3c899d343474f200a6db1cd47fe561777291c02f48ba2dc43"}
Sep 18 18:19:37.192878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.192721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:37.192878 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.192746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt" event={"ID":"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e","Type":"ContainerStarted","Data":"f5bf1e72c730c9815df5aadd5b9cd1ecee8883250524578019432c005e69d69e"}
Sep 18 18:19:37.194105 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.194081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p" event={"ID":"04cf9bfe-97b6-466d-89d3-29b7542a8300","Type":"ContainerStarted","Data":"cd69a4cfc1ef2401a250231634f2e5852dc026ab38a8ab5a4a887bf18d0c7cab"}
Sep 18 18:19:37.194216 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.194123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p" event={"ID":"04cf9bfe-97b6-466d-89d3-29b7542a8300","Type":"ContainerStarted","Data":"0d50183edec90892204ba2aa401d36eb6ac37e61bddb30b6cc51051f11c0ffb3"}
Sep 18 18:19:37.194216 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.194143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:37.208528 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.208484    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt" podStartSLOduration=2.208470264 podStartE2EDuration="2.208470264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:37.206645309 +0000 UTC m=+1147.358520144" watchObservedRunningTime="2026-09-18 18:19:37.208470264 +0000 UTC m=+1147.360345096"
Sep 18 18:19:37.224983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.224935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p" podStartSLOduration=2.224917364 podStartE2EDuration="2.224917364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:37.222849 +0000 UTC m=+1147.374723836" watchObservedRunningTime="2026-09-18 18:19:37.224917364 +0000 UTC m=+1147.376792199"
Sep 18 18:19:37.570649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.570556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:37.910855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:37.910825    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e0c651-dfea-4c75-a64c-f0815b86f287" path="/var/lib/kubelet/pods/d2e0c651-dfea-4c75-a64c-f0815b86f287/volumes"
Sep 18 18:19:38.111970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:38.111936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:38.833826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:38.833781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:39.584368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:39.584331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:44.818718 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:44.818621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:45.365949 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.365911    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"]
Sep 18 18:19:45.366247 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.366221    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" podUID="0e256921-a139-4dd7-b89c-e4bcd45096a2" containerName="maas-api" containerID="cri-o://587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da" gracePeriod=30
Sep 18 18:19:45.394283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.394225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:45.464720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.464491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:45.564136 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.564094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:45.663580 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.663364    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:19:45.670406 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.670374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:45.796896 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.796865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-metrics-certs\") pod \"0e256921-a139-4dd7-b89c-e4bcd45096a2\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") "
Sep 18 18:19:45.797128 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.796978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6256\" (UniqueName: \"kubernetes.io/projected/0e256921-a139-4dd7-b89c-e4bcd45096a2-kube-api-access-q6256\") pod \"0e256921-a139-4dd7-b89c-e4bcd45096a2\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") "
Sep 18 18:19:45.797128 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.797043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e256921-a139-4dd7-b89c-e4bcd45096a2-openshift-service-ca\") pod \"0e256921-a139-4dd7-b89c-e4bcd45096a2\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") "
Sep 18 18:19:45.797266 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.797141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls\") pod \"0e256921-a139-4dd7-b89c-e4bcd45096a2\" (UID: \"0e256921-a139-4dd7-b89c-e4bcd45096a2\") "
Sep 18 18:19:45.797468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.797427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e256921-a139-4dd7-b89c-e4bcd45096a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e256921-a139-4dd7-b89c-e4bcd45096a2" (UID: "0e256921-a139-4dd7-b89c-e4bcd45096a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:19:45.799240 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.799210    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e256921-a139-4dd7-b89c-e4bcd45096a2" (UID: "0e256921-a139-4dd7-b89c-e4bcd45096a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:45.799344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.799235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e256921-a139-4dd7-b89c-e4bcd45096a2" (UID: "0e256921-a139-4dd7-b89c-e4bcd45096a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:45.799344 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.799242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e256921-a139-4dd7-b89c-e4bcd45096a2-kube-api-access-q6256" (OuterVolumeSpecName: "kube-api-access-q6256") pod "0e256921-a139-4dd7-b89c-e4bcd45096a2" (UID: "0e256921-a139-4dd7-b89c-e4bcd45096a2"). InnerVolumeSpecName "kube-api-access-q6256". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:45.898992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.898910    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:45.898992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.898940    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e256921-a139-4dd7-b89c-e4bcd45096a2-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:45.898992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.898950    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6256\" (UniqueName: \"kubernetes.io/projected/0e256921-a139-4dd7-b89c-e4bcd45096a2-kube-api-access-q6256\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:45.898992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.898959    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e256921-a139-4dd7-b89c-e4bcd45096a2-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:45.973617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:45.973563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:46.109770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.109719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:46.261510 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.261472    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e256921-a139-4dd7-b89c-e4bcd45096a2" containerID="587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da" exitCode=0
Sep 18 18:19:46.261677 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.261531    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"
Sep 18 18:19:46.261677 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.261603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" event={"ID":"0e256921-a139-4dd7-b89c-e4bcd45096a2","Type":"ContainerDied","Data":"587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da"}
Sep 18 18:19:46.261677 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.261638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc" event={"ID":"0e256921-a139-4dd7-b89c-e4bcd45096a2","Type":"ContainerDied","Data":"726070abb8a0f5e16fd19bad7ebbd8ea0418e4e69ea8f56f83dc6c0f5ebdae6a"}
Sep 18 18:19:46.261677 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.261658    2580 scope.go:117] "RemoveContainer" containerID="587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da"
Sep 18 18:19:46.270839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.270825    2580 scope.go:117] "RemoveContainer" containerID="587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da"
Sep 18 18:19:46.271134 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:19:46.271105    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da\": container with ID starting with 587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da not found: ID does not exist" containerID="587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da"
Sep 18 18:19:46.271188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.271139    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da"} err="failed to get container status \"587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da\": rpc error: code = NotFound desc = could not find container \"587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da\": container with ID starting with 587085f097ff01dd1f8e787c8831f3e80b7e36d497bdcb84f2ee4bd97edb80da not found: ID does not exist"
Sep 18 18:19:46.276234 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.276212    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"]
Sep 18 18:19:46.279789 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.279769    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7535-d7d6785b8-zbrzc"]
Sep 18 18:19:46.649880 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:46.649788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:47.518649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:47.518609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:47.910633 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:47.910594    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e256921-a139-4dd7-b89c-e4bcd45096a2" path="/var/lib/kubelet/pods/0e256921-a139-4dd7-b89c-e4bcd45096a2/volumes"
Sep 18 18:19:48.201398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:48.201321    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:19:48.201532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:48.201401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:19:48.294648 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:48.294611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:49.112619 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.112585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:49.584557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.584517    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w" podUID="77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" containerName="main" containerID="cri-o://a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1" gracePeriod=45
Sep 18 18:19:49.797163 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.797105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:49.845123 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.845052    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:19:49.937738 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.937697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tmp-dir\") pod \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") "
Sep 18 18:19:49.937738 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.937738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-model-cache\") pod \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") "
Sep 18 18:19:49.937993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.937766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-home\") pod \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") "
Sep 18 18:19:49.937993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.937813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-dshm\") pod \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") "
Sep 18 18:19:49.937993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.937905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjc2v\" (UniqueName: \"kubernetes.io/projected/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-kube-api-access-zjc2v\") pod \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") "
Sep 18 18:19:49.937993 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.937928    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tls-certs\") pod \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\" (UID: \"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e\") "
Sep 18 18:19:49.938231 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.938063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" (UID: "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:49.938231 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.938098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" (UID: "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:49.938231 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.938110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-home" (OuterVolumeSpecName: "home") pod "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" (UID: "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:49.938432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.938370    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:49.938432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.938394    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:49.938432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.938408    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:49.940443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.940419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-kube-api-access-zjc2v" (OuterVolumeSpecName: "kube-api-access-zjc2v") pod "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" (UID: "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e"). InnerVolumeSpecName "kube-api-access-zjc2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:49.940538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.940444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-dshm" (OuterVolumeSpecName: "dshm") pod "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" (UID: "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:49.940538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:49.940498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" (UID: "77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:50.039257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.039221    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:50.039257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.039255    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjc2v\" (UniqueName: \"kubernetes.io/projected/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-kube-api-access-zjc2v\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:50.039257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.039265    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:19:50.291202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.291160    2580 generic.go:356] "Generic (PLEG): container finished" podID="77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" containerID="a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1" exitCode=0
Sep 18 18:19:50.291202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.291200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"
Sep 18 18:19:50.291780 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.291240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w" event={"ID":"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e","Type":"ContainerDied","Data":"a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1"}
Sep 18 18:19:50.291780 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.291288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w" event={"ID":"77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e","Type":"ContainerDied","Data":"d0e32af01b96ca3a7577eef82f1b8adcc6c104171aabf9311a977b4a9ea13f91"}
Sep 18 18:19:50.291780 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.291310    2580 scope.go:117] "RemoveContainer" containerID="a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1"
Sep 18 18:19:50.302813 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.302788    2580 scope.go:117] "RemoveContainer" containerID="a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1"
Sep 18 18:19:50.303127 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:19:50.303105    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1\": container with ID starting with a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1 not found: ID does not exist" containerID="a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1"
Sep 18 18:19:50.303215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.303135    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1"} err="failed to get container status \"a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1\": rpc error: code = NotFound desc = could not find container \"a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1\": container with ID starting with a689c1408f26b8d88228099113b552525a4d37eda6ad7639ad88d8ece22c4ab1 not found: ID does not exist"
Sep 18 18:19:50.312078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.312053    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"]
Sep 18 18:19:50.316178 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.316149    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bcf02f/test-model-bcf02f-kserve-5c647c56c-2w69w"]
Sep 18 18:19:50.607820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:50.607734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:51.185907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:51.185871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:51.859409 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:51.859374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:51.910079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:51.910045    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" path="/var/lib/kubelet/pods/77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e/volumes"
Sep 18 18:19:51.959481 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:51.959453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:52.222179 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.219135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:52.304133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.304097    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5d3295f-369d-4e08-b165-3b94581287eb" containerID="733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0" exitCode=0
Sep 18 18:19:52.304294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.304157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" event={"ID":"f5d3295f-369d-4e08-b165-3b94581287eb","Type":"ContainerDied","Data":"733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"}
Sep 18 18:19:52.304644 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.304626    2580 scope.go:117] "RemoveContainer" containerID="733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"
Sep 18 18:19:52.305755 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.305725    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:52.305836 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.305791    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:52.305836 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.305822    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:52.471585 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.471553    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"]
Sep 18 18:19:52.472246 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.472191    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e256921-a139-4dd7-b89c-e4bcd45096a2" containerName="maas-api"
Sep 18 18:19:52.472246 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.472218    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e256921-a139-4dd7-b89c-e4bcd45096a2" containerName="maas-api"
Sep 18 18:19:52.472370 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.472261    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" containerName="main"
Sep 18 18:19:52.472370 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.472270    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" containerName="main"
Sep 18 18:19:52.472474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.472367    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e256921-a139-4dd7-b89c-e4bcd45096a2" containerName="maas-api"
Sep 18 18:19:52.472474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.472384    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b8dc5a-9fa7-41a5-af28-4d9647ebfa8e" containerName="main"
Sep 18 18:19:52.476064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.476040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.478284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.478262    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-1f7000-openshift-default-dockercfg-wvcj9\""
Sep 18 18:19:52.488910 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.488834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"]
Sep 18 18:19:52.507541 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.507491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:52.565265 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-credential-socket\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565265 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-istio-token\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74829c53-0621-427c-972e-6572dd997284-istio-podinfo\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-socket\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-data\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-certs\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-envoy\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9lw\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-kube-api-access-6h9lw\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.565960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.565780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74829c53-0621-427c-972e-6572dd997284-istiod-ca-cert\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667283 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74829c53-0621-427c-972e-6572dd997284-istiod-ca-cert\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-credential-socket\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-istio-token\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74829c53-0621-427c-972e-6572dd997284-istio-podinfo\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-socket\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-data\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-certs\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667804 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-envoy\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667804 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9lw\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-kube-api-access-6h9lw\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.667957 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-certs\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.668102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.667996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-data\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.668102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.668077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-socket\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.668232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.668156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-credential-socket\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.668293 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.668255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74829c53-0621-427c-972e-6572dd997284-istiod-ca-cert\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.670269 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.670241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-envoy\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.670390 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.670370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74829c53-0621-427c-972e-6572dd997284-istio-podinfo\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.675257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.675233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-istio-token\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.675391 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.675370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9lw\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-kube-api-access-6h9lw\") pod \"e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") " pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.700256 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.700229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:52.793549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.793458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:52.805094 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.805059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:52.882398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.882344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:52.949970 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:52.949931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:53.074721 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.074687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"]
Sep 18 18:19:53.076361 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:19:53.076333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74829c53_0621_427c_972e_6572dd997284.slice/crio-a4bf9c6f674800e4f6e6a8b775a69de1e45bcc09a0a4bda123c950ca9f1cf514 WatchSource:0}: Error finding container a4bf9c6f674800e4f6e6a8b775a69de1e45bcc09a0a4bda123c950ca9f1cf514: Status 404 returned error can't find the container with id a4bf9c6f674800e4f6e6a8b775a69de1e45bcc09a0a4bda123c950ca9f1cf514
Sep 18 18:19:53.078881 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.078849    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:53.078966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.078922    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:53.078966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.078951    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:19:53.313479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.313447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" event={"ID":"f5d3295f-369d-4e08-b165-3b94581287eb","Type":"ContainerStarted","Data":"4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad"}
Sep 18 18:19:53.315534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.315501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" event={"ID":"74829c53-0621-427c-972e-6572dd997284","Type":"ContainerStarted","Data":"8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"}
Sep 18 18:19:53.315750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.315730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" event={"ID":"74829c53-0621-427c-972e-6572dd997284","Type":"ContainerStarted","Data":"a4bf9c6f674800e4f6e6a8b775a69de1e45bcc09a0a4bda123c950ca9f1cf514"}
Sep 18 18:19:53.449210 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.449167    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:53.449625 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.449596    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 18:19:53.449713 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.449649    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 18:19:53.616731 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.616684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:53.794092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:53.793977    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:54.206880 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:54.206837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:54.449103 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:54.449065    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 18:19:54.449292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:54.449143    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 18:19:54.798875 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:54.798840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:54.819408 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:54.819359    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podStartSLOduration=2.819344178 podStartE2EDuration="2.819344178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:53.358990156 +0000 UTC m=+1163.510864995" watchObservedRunningTime="2026-09-18 18:19:54.819344178 +0000 UTC m=+1164.971219014"
Sep 18 18:19:55.002250 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.002202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:55.329975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.329938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:55.342997 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.342963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:19:55.367175 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.367140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:55.449428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.449388    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 18:19:55.449695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.449666    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 18:19:55.457117 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.457082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:55.813874 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:55.813838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:56.449044 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:56.448976    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 18:19:56.449544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:56.449085    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 18:19:56.516288 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:56.516252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:57.252058 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:57.251799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:57.335010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:57.334965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:57.449629 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:57.449578    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 18:19:57.450174 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:57.449674    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 18:19:57.682108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:57.682065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:57.779722 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:57.779681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:58.242081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:58.242043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:58.449331 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:58.449288    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 18:19:58.449507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:58.449373    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 18:19:58.818774 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:58.818735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:59.453077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:59.453049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:59.453357 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:59.453333    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:59.454073 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:59.454056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:19:59.566825 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:19:59.566777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:00.326869 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:00.326834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:03.329871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:03.329832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:03.853836 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:03.853804    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"]
Sep 18 18:20:03.854112 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:03.854066    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" podUID="dd437dfb-6a7e-495a-bb02-95eedec06b28" containerName="maas-api" containerID="cri-o://1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57" gracePeriod=30
Sep 18 18:20:03.885562 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:03.885523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:04.070305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.070264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:04.095695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.095671    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:20:04.170989 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.170957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:04.196138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.196109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls\") pod \"dd437dfb-6a7e-495a-bb02-95eedec06b28\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") "
Sep 18 18:20:04.196292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.196154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd437dfb-6a7e-495a-bb02-95eedec06b28-openshift-service-ca\") pod \"dd437dfb-6a7e-495a-bb02-95eedec06b28\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") "
Sep 18 18:20:04.196292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.196226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwtdh\" (UniqueName: \"kubernetes.io/projected/dd437dfb-6a7e-495a-bb02-95eedec06b28-kube-api-access-wwtdh\") pod \"dd437dfb-6a7e-495a-bb02-95eedec06b28\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") "
Sep 18 18:20:04.196406 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.196351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-metrics-certs\") pod \"dd437dfb-6a7e-495a-bb02-95eedec06b28\" (UID: \"dd437dfb-6a7e-495a-bb02-95eedec06b28\") "
Sep 18 18:20:04.196602 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.196571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd437dfb-6a7e-495a-bb02-95eedec06b28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd437dfb-6a7e-495a-bb02-95eedec06b28" (UID: "dd437dfb-6a7e-495a-bb02-95eedec06b28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:20:04.198615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.198593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd437dfb-6a7e-495a-bb02-95eedec06b28" (UID: "dd437dfb-6a7e-495a-bb02-95eedec06b28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:20:04.198849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.198824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd437dfb-6a7e-495a-bb02-95eedec06b28" (UID: "dd437dfb-6a7e-495a-bb02-95eedec06b28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:20:04.198913 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.198878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd437dfb-6a7e-495a-bb02-95eedec06b28-kube-api-access-wwtdh" (OuterVolumeSpecName: "kube-api-access-wwtdh") pod "dd437dfb-6a7e-495a-bb02-95eedec06b28" (UID: "dd437dfb-6a7e-495a-bb02-95eedec06b28"). InnerVolumeSpecName "kube-api-access-wwtdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:20:04.297888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.297854    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:20:04.297888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.297884    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd437dfb-6a7e-495a-bb02-95eedec06b28-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:20:04.297888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.297893    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd437dfb-6a7e-495a-bb02-95eedec06b28-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:20:04.298174 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.297904    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwtdh\" (UniqueName: \"kubernetes.io/projected/dd437dfb-6a7e-495a-bb02-95eedec06b28-kube-api-access-wwtdh\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:20:04.387477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.387445    2580 generic.go:356] "Generic (PLEG): container finished" podID="dd437dfb-6a7e-495a-bb02-95eedec06b28" containerID="1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57" exitCode=0
Sep 18 18:20:04.387890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.387491    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"
Sep 18 18:20:04.387890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.387533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" event={"ID":"dd437dfb-6a7e-495a-bb02-95eedec06b28","Type":"ContainerDied","Data":"1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57"}
Sep 18 18:20:04.387890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.387581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6" event={"ID":"dd437dfb-6a7e-495a-bb02-95eedec06b28","Type":"ContainerDied","Data":"f65eeae6894d94d274b4d43851355c6ff52ffa33364f10bd024466cd84d4d9b0"}
Sep 18 18:20:04.387890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.387609    2580 scope.go:117] "RemoveContainer" containerID="1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57"
Sep 18 18:20:04.397887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.397866    2580 scope.go:117] "RemoveContainer" containerID="1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57"
Sep 18 18:20:04.398241 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:20:04.398217    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57\": container with ID starting with 1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57 not found: ID does not exist" containerID="1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57"
Sep 18 18:20:04.398310 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.398254    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57"} err="failed to get container status \"1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57\": rpc error: code = NotFound desc = could not find container \"1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57\": container with ID starting with 1731823001ab99ed21811bfc51ea43f7e6ff0c9acf4de44c16e5b6d4d6099b57 not found: ID does not exist"
Sep 18 18:20:04.412323 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.412303    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"]
Sep 18 18:20:04.414428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.414408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcf02f-6c46bb6c99-7pgf6"]
Sep 18 18:20:04.488818 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.488719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:04.588948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:04.588906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:05.526496 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:05.526459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:05.912846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:05.912810    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd437dfb-6a7e-495a-bb02-95eedec06b28" path="/var/lib/kubelet/pods/dd437dfb-6a7e-495a-bb02-95eedec06b28/volumes"
Sep 18 18:20:06.327450 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:06.327365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:06.939092 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:06.939052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:07.497713 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:07.497673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:08.309213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:08.309173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:08.985097 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:08.985062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:09.637814 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:09.637780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:09.979777 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:09.979749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:10.813416 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:10.813382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:10.909932 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:10.909896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:11.063286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.063238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:11.487462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.487431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:11.598099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.598052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:11.892317 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.892237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"]
Sep 18 18:20:11.892744 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.892619    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd437dfb-6a7e-495a-bb02-95eedec06b28" containerName="maas-api"
Sep 18 18:20:11.892744 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.892630    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd437dfb-6a7e-495a-bb02-95eedec06b28" containerName="maas-api"
Sep 18 18:20:11.892744 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.892703    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd437dfb-6a7e-495a-bb02-95eedec06b28" containerName="maas-api"
Sep 18 18:20:11.897188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.897166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.899762 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.899742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-011541-openshift-default-dockercfg-cl4xt\""
Sep 18 18:20:11.913469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.913447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"]
Sep 18 18:20:11.946048 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.945038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:11.990475 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-envoy\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.990669 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-credential-socket\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.990669 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-podinfo\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.990791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpgk\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-kube-api-access-ttpgk\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.990791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a6bc8f-d788-45ad-93f0-1713657e3bac-istiod-ca-cert\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.990791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-certs\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.990791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-token\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.991003 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-data\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:11.991003 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:11.990889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-socket\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.050935 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.050891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:12.092316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-podinfo\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpgk\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-kube-api-access-ttpgk\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a6bc8f-d788-45ad-93f0-1713657e3bac-istiod-ca-cert\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-certs\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-token\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-data\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-socket\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-envoy\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.092883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.092650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-credential-socket\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.093507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.093226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-socket\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.093507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.093267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-certs\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.093507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.093419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-credential-socket\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.093507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.093465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-data\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.093799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.093648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a6bc8f-d788-45ad-93f0-1713657e3bac-istiod-ca-cert\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.095679 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.095436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-podinfo\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.096459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.096435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-envoy\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.103310 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.103290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-token\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.103628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.103605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpgk\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-kube-api-access-ttpgk\") pod \"e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") " pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.210995 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.210956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:12.492564 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.492505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"]
Sep 18 18:20:12.494702 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:20:12.494673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a6bc8f_d788_45ad_93f0_1713657e3bac.slice/crio-37f777138faa91df9376c10cb9e8d868d586e6963fe482fc5eef4f02ac2f6fa3 WatchSource:0}: Error finding container 37f777138faa91df9376c10cb9e8d868d586e6963fe482fc5eef4f02ac2f6fa3: Status 404 returned error can't find the container with id 37f777138faa91df9376c10cb9e8d868d586e6963fe482fc5eef4f02ac2f6fa3
Sep 18 18:20:12.496781 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.496746    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:12.496885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.496821    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:12.496885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:12.496862    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:13.436814 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:13.436772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" event={"ID":"60a6bc8f-d788-45ad-93f0-1713657e3bac","Type":"ContainerStarted","Data":"03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"}
Sep 18 18:20:13.437242 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:13.436840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" event={"ID":"60a6bc8f-d788-45ad-93f0-1713657e3bac","Type":"ContainerStarted","Data":"37f777138faa91df9376c10cb9e8d868d586e6963fe482fc5eef4f02ac2f6fa3"}
Sep 18 18:20:13.457750 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:13.457694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podStartSLOduration=2.45767402 podStartE2EDuration="2.45767402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:13.457007308 +0000 UTC m=+1183.608882143" watchObservedRunningTime="2026-09-18 18:20:13.45767402 +0000 UTC m=+1183.609548874"
Sep 18 18:20:14.211670 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:14.211635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:14.216895 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:14.216871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:14.442850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:14.442805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:14.443647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:14.443623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:14.733554 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:14.733520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:15.448955 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.448927    2580 generic.go:356] "Generic (PLEG): container finished" podID="74829c53-0621-427c-972e-6572dd997284" containerID="8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3" exitCode=0
Sep 18 18:20:15.449334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.449001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" event={"ID":"74829c53-0621-427c-972e-6572dd997284","Type":"ContainerDied","Data":"8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"}
Sep 18 18:20:15.449552 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.449532    2580 scope.go:117] "RemoveContainer" containerID="8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"
Sep 18 18:20:15.450794 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.450764    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:15.450876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.450832    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:15.450876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.450863    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:15.611071 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.610960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:15.710415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:15.710381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:16.437684 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:16.437565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:16.456897 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:16.456863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" event={"ID":"74829c53-0621-427c-972e-6572dd997284","Type":"ContainerStarted","Data":"03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b"}
Sep 18 18:20:16.794513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:16.794437    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:20:16.794652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:16.794609    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:20:16.794695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:16.794654    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:20:17.555853 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:17.555808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:17.794797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:17.794762    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:20:17.794982 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:17.794818    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:20:18.124086 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:18.124018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:18.685519 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:18.685483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:18.746040 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:18.745994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:18.794715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:18.794680    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:20:18.794887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:18.794745    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:20:19.031353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.031243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:19.179296 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.179260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:19.279613 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.279567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:19.360979 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.360863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:19.483106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.483074    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"]
Sep 18 18:20:19.486579 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.486558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.488563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.488542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-011541\""
Sep 18 18:20:19.488704 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.488686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-011541-dockercfg-zwdvc\""
Sep 18 18:20:19.497764 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.497742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"]
Sep 18 18:20:19.581838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.581800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdmms\" (UniqueName: \"kubernetes.io/projected/be554f45-681e-4bea-bfa5-06ea1169936f-kube-api-access-cdmms\") pod \"payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.582017 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.581895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be554f45-681e-4bea-bfa5-06ea1169936f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.617183 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.617103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:19.682818 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.682788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdmms\" (UniqueName: \"kubernetes.io/projected/be554f45-681e-4bea-bfa5-06ea1169936f-kube-api-access-cdmms\") pod \"payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.683010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.682899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be554f45-681e-4bea-bfa5-06ea1169936f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.683891 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.683864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be554f45-681e-4bea-bfa5-06ea1169936f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.698245 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.698218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdmms\" (UniqueName: \"kubernetes.io/projected/be554f45-681e-4bea-bfa5-06ea1169936f-kube-api-access-cdmms\") pod \"payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.794506 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.794471    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:20:19.794680 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.794536    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:20:19.798640 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.798620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:19.867739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.867653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:19.951391 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:19.951347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:20.110255 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.110222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"]
Sep 18 18:20:20.113084 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:20:20.113049    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe554f45_681e_4bea_bfa5_06ea1169936f.slice/crio-9069f2f82aa73501d3024145d2fa8af4c38b28cf6508ff51dc6c231a1584d8c5 WatchSource:0}: Error finding container 9069f2f82aa73501d3024145d2fa8af4c38b28cf6508ff51dc6c231a1584d8c5: Status 404 returned error can't find the container with id 9069f2f82aa73501d3024145d2fa8af4c38b28cf6508ff51dc6c231a1584d8c5
Sep 18 18:20:20.479493 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.479442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb" event={"ID":"be554f45-681e-4bea-bfa5-06ea1169936f","Type":"ContainerStarted","Data":"60bcb86fd0731bbca224f770500e6b5119c3728ddf4eb5dde62248bac13b7193"}
Sep 18 18:20:20.479739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.479510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb" event={"ID":"be554f45-681e-4bea-bfa5-06ea1169936f","Type":"ContainerStarted","Data":"9069f2f82aa73501d3024145d2fa8af4c38b28cf6508ff51dc6c231a1584d8c5"}
Sep 18 18:20:20.479739 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.479666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:20.496237 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.496191    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb" podStartSLOduration=1.496176339 podStartE2EDuration="1.496176339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:20.494981669 +0000 UTC m=+1190.646856504" watchObservedRunningTime="2026-09-18 18:20:20.496176339 +0000 UTC m=+1190.648051172"
Sep 18 18:20:20.699683 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.699647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:20.794858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.794776    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:20:20.794858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:20.794836    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:20:21.794132 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:21.794098    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:20:21.794491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:21.794154    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:20:21.881921 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:21.881878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:22.580076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:22.580015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:22.652286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:22.652253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:22.798443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:22.798408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:20:22.798821 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:22.798802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:20:22.799574 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:22.799557    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:20:22.921076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:22.921047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:23.663721 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:23.663685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:24.348064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:24.348014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:25.064688 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:25.064653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:25.590286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:25.590255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:26.418004 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:26.417950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:27.674393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.674362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:27.774728 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.774683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:27.854057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.854004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:27.892557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.892520    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"]
Sep 18 18:20:27.897078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.897057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:27.899208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.899187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6fa1a2a2-dockercfg-lgntz\""
Sep 18 18:20:27.899317 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.899192    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6fa1a2a2\""
Sep 18 18:20:27.913062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.913004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"]
Sep 18 18:20:27.968932 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.968845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1feb3-d737-40b3-9e19-85d841df3965-plugins-config-volume\") pod \"payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") " pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:27.968932 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:27.968917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcp97\" (UniqueName: \"kubernetes.io/projected/f9c1feb3-d737-40b3-9e19-85d841df3965-kube-api-access-gcp97\") pod \"payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") " pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:28.070593 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.070557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1feb3-d737-40b3-9e19-85d841df3965-plugins-config-volume\") pod \"payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") " pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:28.070593 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.070599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcp97\" (UniqueName: \"kubernetes.io/projected/f9c1feb3-d737-40b3-9e19-85d841df3965-kube-api-access-gcp97\") pod \"payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") " pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:28.071304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.071283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1feb3-d737-40b3-9e19-85d841df3965-plugins-config-volume\") pod \"payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") " pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:28.078473 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.078454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcp97\" (UniqueName: \"kubernetes.io/projected/f9c1feb3-d737-40b3-9e19-85d841df3965-kube-api-access-gcp97\") pod \"payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") " pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:28.212979 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.212948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:28.485549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.485525    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"]
Sep 18 18:20:28.487284 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:20:28.487243    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c1feb3_d737_40b3_9e19_85d841df3965.slice/crio-6f3b72cdf2a19dc0eeb8bea2a35224a860bb0f06913dcb4a5222e8fa9c008eb4 WatchSource:0}: Error finding container 6f3b72cdf2a19dc0eeb8bea2a35224a860bb0f06913dcb4a5222e8fa9c008eb4: Status 404 returned error can't find the container with id 6f3b72cdf2a19dc0eeb8bea2a35224a860bb0f06913dcb4a5222e8fa9c008eb4
Sep 18 18:20:28.523732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:28.523695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb" event={"ID":"f9c1feb3-d737-40b3-9e19-85d841df3965","Type":"ContainerStarted","Data":"6f3b72cdf2a19dc0eeb8bea2a35224a860bb0f06913dcb4a5222e8fa9c008eb4"}
Sep 18 18:20:29.534052 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:29.533990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb" event={"ID":"f9c1feb3-d737-40b3-9e19-85d841df3965","Type":"ContainerStarted","Data":"048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567"}
Sep 18 18:20:29.534916 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:29.534889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:29.549221 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:29.549079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb" podStartSLOduration=2.549057633 podStartE2EDuration="2.549057633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:29.548633139 +0000 UTC m=+1199.700507974" watchObservedRunningTime="2026-09-18 18:20:29.549057633 +0000 UTC m=+1199.700932472"
Sep 18 18:20:31.486804 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:31.486774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:20:38.590919 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:38.590882    2580 generic.go:356] "Generic (PLEG): container finished" podID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerID="03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f" exitCode=0
Sep 18 18:20:38.591427 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:38.591044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" event={"ID":"60a6bc8f-d788-45ad-93f0-1713657e3bac","Type":"ContainerDied","Data":"03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"}
Sep 18 18:20:38.591500 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:38.591467    2580 scope.go:117] "RemoveContainer" containerID="03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"
Sep 18 18:20:38.592617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:38.592585    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:38.592715 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:38.592671    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:38.592777 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:38.592722    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:20:39.597881 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:39.597844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" event={"ID":"60a6bc8f-d788-45ad-93f0-1713657e3bac","Type":"ContainerStarted","Data":"171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6"}
Sep 18 18:20:39.835863 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:39.835815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:40.211459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:40.211421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:40.212480 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:40.212456    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 18 18:20:40.212590 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:40.212525    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 18 18:20:41.136154 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:41.136106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:41.211571 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:41.211542    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 18 18:20:41.211738 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:41.211607    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 18 18:20:41.555410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:41.555376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:20:41.908268 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:41.908236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:42.211299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:42.211226    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 18 18:20:42.211712 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:42.211295    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 18 18:20:42.602105 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:42.602002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:43.211986 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:43.211950    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 18 18:20:43.212415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:43.212004    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 18 18:20:43.334795 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:43.334696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:44.117969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:44.117933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:44.211866 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:44.211834    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 18 18:20:44.212066 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:44.211887    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 18 18:20:44.443555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:44.443516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:44.917081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:44.917048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:45.216290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:45.216217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:45.217218 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:45.217201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:20:45.484218 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:45.484134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:45.594571 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:45.594527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:45.808134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:45.808055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:46.103342 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:46.103260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:46.312524 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:46.312496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:46.418831 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:46.418794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:47.117511 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:47.117475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:47.523502 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:47.523465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:47.978463 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:47.978432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:48.655730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:48.655697    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:48.761253 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:48.761218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:48.857581 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:48.857544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:49.524446 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:49.524410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:50.234764 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.234730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:50.304805 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.304764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:50.611545 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.611455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:50.713776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.713745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:50.814636 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.814601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:50.826583 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.826553    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"]
Sep 18 18:20:50.830672 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.830655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:50.833271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.833253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-99602c-dockercfg-nnfjw\""
Sep 18 18:20:50.833484 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.833468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-99602c\""
Sep 18 18:20:50.838589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.838567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"]
Sep 18 18:20:50.938894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.938810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fc15ee-34ca-4f87-a743-c845accfa5d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:50.938894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:50.938872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgbn\" (UniqueName: \"kubernetes.io/projected/21fc15ee-34ca-4f87-a743-c845accfa5d2-kube-api-access-zjgbn\") pod \"payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.040286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.040243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fc15ee-34ca-4f87-a743-c845accfa5d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.040468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.040304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgbn\" (UniqueName: \"kubernetes.io/projected/21fc15ee-34ca-4f87-a743-c845accfa5d2-kube-api-access-zjgbn\") pod \"payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.041249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.041214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fc15ee-34ca-4f87-a743-c845accfa5d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.052290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.052265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgbn\" (UniqueName: \"kubernetes.io/projected/21fc15ee-34ca-4f87-a743-c845accfa5d2-kube-api-access-zjgbn\") pod \"payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.142455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.142421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.331052 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.330986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:51.429193 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.429163    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"]
Sep 18 18:20:51.430691 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:20:51.430662    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fc15ee_34ca_4f87_a743_c845accfa5d2.slice/crio-43d715579040244b1d153e29f183b0c6f196a81214783923ba34ef2bd1676e13 WatchSource:0}: Error finding container 43d715579040244b1d153e29f183b0c6f196a81214783923ba34ef2bd1676e13: Status 404 returned error can't find the container with id 43d715579040244b1d153e29f183b0c6f196a81214783923ba34ef2bd1676e13
Sep 18 18:20:51.659955 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.659915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv" event={"ID":"21fc15ee-34ca-4f87-a743-c845accfa5d2","Type":"ContainerStarted","Data":"01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978"}
Sep 18 18:20:51.660161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.659978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv" event={"ID":"21fc15ee-34ca-4f87-a743-c845accfa5d2","Type":"ContainerStarted","Data":"43d715579040244b1d153e29f183b0c6f196a81214783923ba34ef2bd1676e13"}
Sep 18 18:20:51.660161 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.660077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:20:51.675926 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:51.675868    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv" podStartSLOduration=1.675849007 podStartE2EDuration="1.675849007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:51.674627497 +0000 UTC m=+1221.826502333" watchObservedRunningTime="2026-09-18 18:20:51.675849007 +0000 UTC m=+1221.827723843"
Sep 18 18:20:52.239312 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:52.239260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:52.742428 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:52.742394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:53.527109 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:53.527068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:54.219387 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:54.219354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:55.008902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:20:55.008860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:02.667629 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:02.667596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:21:12.391192 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:12.391149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:12.967602 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:12.967562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:13.623874 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:13.623840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:13.723972 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:13.723932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:14.425474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:14.425425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:15.202834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:15.202799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:15.931587 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:15.931556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:16.791833 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:16.791792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:17.498058 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:17.498004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:19.228098 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:19.228054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:19.279271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:19.279234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:26.070116 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.070078    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"]
Sep 18 18:21:26.073673 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.073656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.076494 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.076474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6fa1a2a2\"/\"e2e-lifecycle-model-6fa1a2a2-kserve-self-signed-certs\""
Sep 18 18:21:26.076635 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.076476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6fa1a2a2\"/\"default-dockercfg-jhhjh\""
Sep 18 18:21:26.076635 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.076474    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6fa1a2a2\"/\"kube-root-ca.crt\""
Sep 18 18:21:26.076635 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.076521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6fa1a2a2\"/\"openshift-service-ca.crt\""
Sep 18 18:21:26.082633 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.082609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"]
Sep 18 18:21:26.180861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.180827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-tmp-dir\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.181074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.180912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9mq\" (UniqueName: \"kubernetes.io/projected/dede6da5-7e4f-40aa-9601-08827f72f516-kube-api-access-zx9mq\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.181074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.180973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-model-cache\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.181074 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.181054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-dshm\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.181254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.181097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dede6da5-7e4f-40aa-9601-08827f72f516-tls-certs\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.181254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.181155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-home\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.184156 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.184126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:26.282354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-tmp-dir\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9mq\" (UniqueName: \"kubernetes.io/projected/dede6da5-7e4f-40aa-9601-08827f72f516-kube-api-access-zx9mq\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-model-cache\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-dshm\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282751 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dede6da5-7e4f-40aa-9601-08827f72f516-tls-certs\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-home\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-tmp-dir\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.282807 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.282798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-model-cache\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.283090 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.283066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-home\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.284887 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.284866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-dshm\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.285443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.285420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dede6da5-7e4f-40aa-9601-08827f72f516-tls-certs\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.290794 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.290772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9mq\" (UniqueName: \"kubernetes.io/projected/dede6da5-7e4f-40aa-9601-08827f72f516-kube-api-access-zx9mq\") pod \"e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") " pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.383553 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.383478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:26.663244 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.663215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"]
Sep 18 18:21:26.664582 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:21:26.664556    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddede6da5_7e4f_40aa_9601_08827f72f516.slice/crio-c6f69473cc412591f717395b7447c40b1e78cb3d05835d7443e3c3c9d16365c0 WatchSource:0}: Error finding container c6f69473cc412591f717395b7447c40b1e78cb3d05835d7443e3c3c9d16365c0: Status 404 returned error can't find the container with id c6f69473cc412591f717395b7447c40b1e78cb3d05835d7443e3c3c9d16365c0
Sep 18 18:21:26.847631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.847594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn" event={"ID":"dede6da5-7e4f-40aa-9601-08827f72f516","Type":"ContainerStarted","Data":"e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183"}
Sep 18 18:21:26.847631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.847632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn" event={"ID":"dede6da5-7e4f-40aa-9601-08827f72f516","Type":"ContainerStarted","Data":"c6f69473cc412591f717395b7447c40b1e78cb3d05835d7443e3c3c9d16365c0"}
Sep 18 18:21:26.864647 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:26.864593    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn" podStartSLOduration=0.864578383 podStartE2EDuration="864.578383ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:21:26.86240587 +0000 UTC m=+1257.014280719" watchObservedRunningTime="2026-09-18 18:21:26.864578383 +0000 UTC m=+1257.016453273"
Sep 18 18:21:32.311586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:32.311540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:32.819537 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:32.819498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:33.635221 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:33.635186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:34.513930 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:34.513885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:35.319279 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:35.319241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:35.997527 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:35.997496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:36.383965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:36.383883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:36.383965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:36.383930    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:36.396477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:36.396457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:36.646911 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:36.646818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:36.910400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:36.910374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:21:37.308687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:37.308602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"]
Sep 18 18:21:37.353139 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:37.353092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:37.446575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:37.446530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:38.073918 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:38.073884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:38.623387 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:38.623343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:47.935672 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:47.935627    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"]
Sep 18 18:21:47.936160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:47.935877    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q" podUID="84a2da44-97d4-47fa-b6ee-e7763deb8617" containerName="maas-api" containerID="cri-o://172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567" gracePeriod=30
Sep 18 18:21:47.966154 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:47.966102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:48.039248 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.039204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:48.204065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.203943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:48.278282 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.278251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:48.409532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.409509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:21:48.497819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.497738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a2da44-97d4-47fa-b6ee-e7763deb8617-openshift-service-ca\") pod \"84a2da44-97d4-47fa-b6ee-e7763deb8617\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") "
Sep 18 18:21:48.497819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.497783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-metrics-certs\") pod \"84a2da44-97d4-47fa-b6ee-e7763deb8617\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") "
Sep 18 18:21:48.498021 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.497832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-maas-api-tls\") pod \"84a2da44-97d4-47fa-b6ee-e7763deb8617\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") "
Sep 18 18:21:48.498021 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.497913    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxqq7\" (UniqueName: \"kubernetes.io/projected/84a2da44-97d4-47fa-b6ee-e7763deb8617-kube-api-access-zxqq7\") pod \"84a2da44-97d4-47fa-b6ee-e7763deb8617\" (UID: \"84a2da44-97d4-47fa-b6ee-e7763deb8617\") "
Sep 18 18:21:48.498193 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.498166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a2da44-97d4-47fa-b6ee-e7763deb8617-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84a2da44-97d4-47fa-b6ee-e7763deb8617" (UID: "84a2da44-97d4-47fa-b6ee-e7763deb8617"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:21:48.500126 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.500090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84a2da44-97d4-47fa-b6ee-e7763deb8617" (UID: "84a2da44-97d4-47fa-b6ee-e7763deb8617"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:21:48.500267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.500150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84a2da44-97d4-47fa-b6ee-e7763deb8617" (UID: "84a2da44-97d4-47fa-b6ee-e7763deb8617"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:21:48.500267 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.500212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a2da44-97d4-47fa-b6ee-e7763deb8617-kube-api-access-zxqq7" (OuterVolumeSpecName: "kube-api-access-zxqq7") pod "84a2da44-97d4-47fa-b6ee-e7763deb8617" (UID: "84a2da44-97d4-47fa-b6ee-e7763deb8617"). InnerVolumeSpecName "kube-api-access-zxqq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:21:48.599600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.599563    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:48.599600 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.599598    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a2da44-97d4-47fa-b6ee-e7763deb8617-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:48.599832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.599614    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqq7\" (UniqueName: \"kubernetes.io/projected/84a2da44-97d4-47fa-b6ee-e7763deb8617-kube-api-access-zxqq7\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:48.599832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.599627    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a2da44-97d4-47fa-b6ee-e7763deb8617-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:48.691599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.691563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:48.814761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.814656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:48.963488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.963452    2580 generic.go:356] "Generic (PLEG): container finished" podID="84a2da44-97d4-47fa-b6ee-e7763deb8617" containerID="172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567" exitCode=0
Sep 18 18:21:48.963872 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.963498    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"
Sep 18 18:21:48.963872 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.963542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q" event={"ID":"84a2da44-97d4-47fa-b6ee-e7763deb8617","Type":"ContainerDied","Data":"172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567"}
Sep 18 18:21:48.963872 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.963591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q" event={"ID":"84a2da44-97d4-47fa-b6ee-e7763deb8617","Type":"ContainerDied","Data":"cd0b5386a512222acb334a7719ad3eeca825c3714a04fd10d029389e55ff9bfe"}
Sep 18 18:21:48.963872 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.963614    2580 scope.go:117] "RemoveContainer" containerID="172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567"
Sep 18 18:21:48.973455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.973430    2580 scope.go:117] "RemoveContainer" containerID="172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567"
Sep 18 18:21:48.973699 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:21:48.973681    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567\": container with ID starting with 172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567 not found: ID does not exist" containerID="172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567"
Sep 18 18:21:48.973743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.973707    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567"} err="failed to get container status \"172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567\": rpc error: code = NotFound desc = could not find container \"172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567\": container with ID starting with 172302183a62e70dfd424ad5ab8570615028b2be5ca05bcfaff30f35dd002567 not found: ID does not exist"
Sep 18 18:21:48.984421 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.984401    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"]
Sep 18 18:21:48.990473 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:48.990450    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-039dbc-7b654cc6c8-7rl9q"]
Sep 18 18:21:49.681588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:49.681552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:49.910925 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:49.910889    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a2da44-97d4-47fa-b6ee-e7763deb8617" path="/var/lib/kubelet/pods/84a2da44-97d4-47fa-b6ee-e7763deb8617/volumes"
Sep 18 18:21:50.464588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:50.464553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:51.013531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:51.013497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:51.689623 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:51.689569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:51.790901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:51.790854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:52.329140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.329078    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54" podUID="90b2ebbb-8ddc-4e24-ac16-79655fefdecb" containerName="main" containerID="cri-o://af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4" gracePeriod=45
Sep 18 18:21:52.400158 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.400120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:52.590962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.590906    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:21:52.641466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-model-cache\") pod \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") "
Sep 18 18:21:52.641634 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-home\") pod \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") "
Sep 18 18:21:52.641702 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tmp-dir\") pod \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") "
Sep 18 18:21:52.641760 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-model-cache" (OuterVolumeSpecName: "model-cache") pod "90b2ebbb-8ddc-4e24-ac16-79655fefdecb" (UID: "90b2ebbb-8ddc-4e24-ac16-79655fefdecb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:21:52.641760 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzcm8\" (UniqueName: \"kubernetes.io/projected/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-kube-api-access-xzcm8\") pod \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") "
Sep 18 18:21:52.641877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tls-certs\") pod \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") "
Sep 18 18:21:52.641877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-home" (OuterVolumeSpecName: "home") pod "90b2ebbb-8ddc-4e24-ac16-79655fefdecb" (UID: "90b2ebbb-8ddc-4e24-ac16-79655fefdecb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:21:52.641877 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-dshm\") pod \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\" (UID: \"90b2ebbb-8ddc-4e24-ac16-79655fefdecb\") "
Sep 18 18:21:52.642073 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.641967    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90b2ebbb-8ddc-4e24-ac16-79655fefdecb" (UID: "90b2ebbb-8ddc-4e24-ac16-79655fefdecb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:21:52.642328 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.642273    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:52.642328 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.642294    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:52.642328 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.642305    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:52.644792 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.644713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-dshm" (OuterVolumeSpecName: "dshm") pod "90b2ebbb-8ddc-4e24-ac16-79655fefdecb" (UID: "90b2ebbb-8ddc-4e24-ac16-79655fefdecb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:21:52.644792 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.644722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90b2ebbb-8ddc-4e24-ac16-79655fefdecb" (UID: "90b2ebbb-8ddc-4e24-ac16-79655fefdecb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:21:52.644792 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.644755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-kube-api-access-xzcm8" (OuterVolumeSpecName: "kube-api-access-xzcm8") pod "90b2ebbb-8ddc-4e24-ac16-79655fefdecb" (UID: "90b2ebbb-8ddc-4e24-ac16-79655fefdecb"). InnerVolumeSpecName "kube-api-access-xzcm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:21:52.743107 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.743074    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzcm8\" (UniqueName: \"kubernetes.io/projected/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-kube-api-access-xzcm8\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:52.743107 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.743102    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:52.743107 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.743113    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b2ebbb-8ddc-4e24-ac16-79655fefdecb-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:21:52.987268 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.987234    2580 generic.go:356] "Generic (PLEG): container finished" podID="90b2ebbb-8ddc-4e24-ac16-79655fefdecb" containerID="af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4" exitCode=0
Sep 18 18:21:52.987432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.987278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"
Sep 18 18:21:52.987432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.987285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54" event={"ID":"90b2ebbb-8ddc-4e24-ac16-79655fefdecb","Type":"ContainerDied","Data":"af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4"}
Sep 18 18:21:52.987432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.987319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54" event={"ID":"90b2ebbb-8ddc-4e24-ac16-79655fefdecb","Type":"ContainerDied","Data":"7faeb316928780cd4d89e892fd2d5ac6ae572306bb1a7937dd8c2a57a28a19dd"}
Sep 18 18:21:52.987432 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.987335    2580 scope.go:117] "RemoveContainer" containerID="af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4"
Sep 18 18:21:52.996725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.996705    2580 scope.go:117] "RemoveContainer" containerID="af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4"
Sep 18 18:21:52.996977 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:21:52.996954    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4\": container with ID starting with af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4 not found: ID does not exist" containerID="af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4"
Sep 18 18:21:52.997090 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:52.996984    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4"} err="failed to get container status \"af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4\": rpc error: code = NotFound desc = could not find container \"af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4\": container with ID starting with af8fe345bd7c3e3bc1a4ce9cb9ee9ed130ae3636457ef7855786b0abb71433c4 not found: ID does not exist"
Sep 18 18:21:53.007474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:53.007452    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"]
Sep 18 18:21:53.010781 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:53.010762    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-039dbc/premium-w3-039dbc-kserve-77875d57b8-rlj54"]
Sep 18 18:21:53.218015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:53.217964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:53.912139 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:53.912109    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b2ebbb-8ddc-4e24-ac16-79655fefdecb" path="/var/lib/kubelet/pods/90b2ebbb-8ddc-4e24-ac16-79655fefdecb/volumes"
Sep 18 18:21:54.356285 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:54.356206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:54.460611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:54.460577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:54.623410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:54.623324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"]
Sep 18 18:21:54.624096 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:54.624063    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" containerID="cri-o://21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb" gracePeriod=30
Sep 18 18:21:54.647345 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:54.647317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:55.696985 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:55.696942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:55.750835 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:55.750793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:56.221248 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:56.221213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:59.275700 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:59.275631    2580 patch_prober.go:28] interesting pod/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 18:21:59.275700 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:59.275682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 18:21:59.890494 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:21:59.890468    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:22:00.011394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-token\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-podinfo\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq9ql\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-kube-api-access-qq9ql\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-certs\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istiod-ca-cert\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-data\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011851 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-envoy\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011851 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-socket\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011851 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-credential-socket\") pod \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\" (UID: \"00ec0aca-3d3a-4a25-9d71-e80034f52d0a\") "
Sep 18 18:22:00.011992 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.011865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:00.012192 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.012063    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.012192 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.012088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:00.012192 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.012149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:00.012441 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.012308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:22:00.012501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.012440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:00.013743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.013714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:22:00.013852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.013781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-kube-api-access-qq9ql" (OuterVolumeSpecName: "kube-api-access-qq9ql") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "kube-api-access-qq9ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:22:00.013852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.013795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:22:00.014020 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.014006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00ec0aca-3d3a-4a25-9d71-e80034f52d0a" (UID: "00ec0aca-3d3a-4a25-9d71-e80034f52d0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:00.026713 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.026679    2580 generic.go:356] "Generic (PLEG): container finished" podID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerID="21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb" exitCode=0
Sep 18 18:22:00.026822 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.026747    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"
Sep 18 18:22:00.026822 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.026766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" event={"ID":"00ec0aca-3d3a-4a25-9d71-e80034f52d0a","Type":"ContainerDied","Data":"21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb"}
Sep 18 18:22:00.026901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.026823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l" event={"ID":"00ec0aca-3d3a-4a25-9d71-e80034f52d0a","Type":"ContainerDied","Data":"93692ad7cc502721be8f4846ca0ba5a026eb3a61b657adef4908fe20011e2f3c"}
Sep 18 18:22:00.026901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.026845    2580 scope.go:117] "RemoveContainer" containerID="21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb"
Sep 18 18:22:00.037367 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.037351    2580 scope.go:117] "RemoveContainer" containerID="cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"
Sep 18 18:22:00.045163 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.045145    2580 scope.go:117] "RemoveContainer" containerID="21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb"
Sep 18 18:22:00.045503 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:22:00.045466    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb\": container with ID starting with 21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb not found: ID does not exist" containerID="21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb"
Sep 18 18:22:00.045584 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.045503    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb"} err="failed to get container status \"21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb\": rpc error: code = NotFound desc = could not find container \"21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb\": container with ID starting with 21a831b1a5491b2053f6649262f6239edf3cb9d45ccfd124a1eda95900d642eb not found: ID does not exist"
Sep 18 18:22:00.045584 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.045520    2580 scope.go:117] "RemoveContainer" containerID="cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"
Sep 18 18:22:00.046415 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:22:00.046392    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631\": container with ID starting with cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631 not found: ID does not exist" containerID="cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"
Sep 18 18:22:00.046501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.046425    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631"} err="failed to get container status \"cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631\": rpc error: code = NotFound desc = could not find container \"cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631\": container with ID starting with cb3cb90bc71250c9adda9ab7c9ad6ff950e6927ebcba21b5b789a4b873a80631 not found: ID does not exist"
Sep 18 18:22:00.048649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.048629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"]
Sep 18 18:22:00.053254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.053231    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-039dbc-openshift-default-78c6c5b876-gmc7l"]
Sep 18 18:22:00.112786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112757    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112782    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112795    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112807    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112819    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112832    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112845    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:00.112923 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:00.112857    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9ql\" (UniqueName: \"kubernetes.io/projected/00ec0aca-3d3a-4a25-9d71-e80034f52d0a-kube-api-access-qq9ql\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:01.911411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:01.911376    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" path="/var/lib/kubelet/pods/00ec0aca-3d3a-4a25-9d71-e80034f52d0a/volumes"
Sep 18 18:22:14.329885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:14.329846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:14.392368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:14.392314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:14.914081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:14.914037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:20.565761 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:20.565713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:24.441850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:24.441810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:25.046684 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:25.046651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:25.705427 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:25.705395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:26.522560 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:26.522523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:27.203377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:27.203337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:27.831337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:27.831294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:28.621596 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:28.621548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:31.581769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:31.581735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:39.306671 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.306634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:39.325823 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.324644    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"]
Sep 18 18:22:39.325823 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.324918    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb" podUID="f9c1feb3-d737-40b3-9e19-85d841df3965" containerName="payload-processing" containerID="cri-o://048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567" gracePeriod=30
Sep 18 18:22:39.406984 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.406944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:39.508548 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.508488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:39.582311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.582270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:39.600080 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.600054    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:22:39.671219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.671187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1feb3-d737-40b3-9e19-85d841df3965-plugins-config-volume\") pod \"f9c1feb3-d737-40b3-9e19-85d841df3965\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") "
Sep 18 18:22:39.671392 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.671239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcp97\" (UniqueName: \"kubernetes.io/projected/f9c1feb3-d737-40b3-9e19-85d841df3965-kube-api-access-gcp97\") pod \"f9c1feb3-d737-40b3-9e19-85d841df3965\" (UID: \"f9c1feb3-d737-40b3-9e19-85d841df3965\") "
Sep 18 18:22:39.671602 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.671578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c1feb3-d737-40b3-9e19-85d841df3965-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9c1feb3-d737-40b3-9e19-85d841df3965" (UID: "f9c1feb3-d737-40b3-9e19-85d841df3965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:22:39.673398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.673378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c1feb3-d737-40b3-9e19-85d841df3965-kube-api-access-gcp97" (OuterVolumeSpecName: "kube-api-access-gcp97") pod "f9c1feb3-d737-40b3-9e19-85d841df3965" (UID: "f9c1feb3-d737-40b3-9e19-85d841df3965"). InnerVolumeSpecName "kube-api-access-gcp97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:22:39.772480 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.772440    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcp97\" (UniqueName: \"kubernetes.io/projected/f9c1feb3-d737-40b3-9e19-85d841df3965-kube-api-access-gcp97\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:39.772480 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.772479    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1feb3-d737-40b3-9e19-85d841df3965-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:22:39.989825 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:39.989788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:40.239143 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.239107    2580 generic.go:356] "Generic (PLEG): container finished" podID="f9c1feb3-d737-40b3-9e19-85d841df3965" containerID="048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567" exitCode=0
Sep 18 18:22:40.239306 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.239159    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"
Sep 18 18:22:40.239306 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.239186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb" event={"ID":"f9c1feb3-d737-40b3-9e19-85d841df3965","Type":"ContainerDied","Data":"048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567"}
Sep 18 18:22:40.239306 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.239238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb" event={"ID":"f9c1feb3-d737-40b3-9e19-85d841df3965","Type":"ContainerDied","Data":"6f3b72cdf2a19dc0eeb8bea2a35224a860bb0f06913dcb4a5222e8fa9c008eb4"}
Sep 18 18:22:40.239306 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.239260    2580 scope.go:117] "RemoveContainer" containerID="048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567"
Sep 18 18:22:40.248250 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.248233    2580 scope.go:117] "RemoveContainer" containerID="048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567"
Sep 18 18:22:40.248481 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:22:40.248466    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567\": container with ID starting with 048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567 not found: ID does not exist" containerID="048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567"
Sep 18 18:22:40.248537 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.248488    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567"} err="failed to get container status \"048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567\": rpc error: code = NotFound desc = could not find container \"048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567\": container with ID starting with 048766add4fa96cdb4c9d1a6cd08668a7dc73d6294ce6a6c912b90b3ae55c567 not found: ID does not exist"
Sep 18 18:22:40.253929 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.253908    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"]
Sep 18 18:22:40.257322 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.257292    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6fa1a2a2-567b7b98b5-pjngb"]
Sep 18 18:22:40.655098 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:40.655000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:41.309133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:41.309105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:41.911437 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:41.911405    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c1feb3-d737-40b3-9e19-85d841df3965" path="/var/lib/kubelet/pods/f9c1feb3-d737-40b3-9e19-85d841df3965/volumes"
Sep 18 18:22:42.116508 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:42.116476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:42.622712 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:42.622622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:42.803059 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:42.803001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:43.482849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:43.482809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:47.609960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:47.609926    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"]
Sep 18 18:22:47.704358 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:22:47.704305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:02.222215 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.222182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:02.630137 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.630006    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn" podUID="dede6da5-7e4f-40aa-9601-08827f72f516" containerName="main" containerID="cri-o://e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183" gracePeriod=45
Sep 18 18:23:02.879339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.879312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:23:02.976587 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx9mq\" (UniqueName: \"kubernetes.io/projected/dede6da5-7e4f-40aa-9601-08827f72f516-kube-api-access-zx9mq\") pod \"dede6da5-7e4f-40aa-9601-08827f72f516\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") "
Sep 18 18:23:02.976765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976622    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-home\") pod \"dede6da5-7e4f-40aa-9601-08827f72f516\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") "
Sep 18 18:23:02.976765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-tmp-dir\") pod \"dede6da5-7e4f-40aa-9601-08827f72f516\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") "
Sep 18 18:23:02.976765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dede6da5-7e4f-40aa-9601-08827f72f516-tls-certs\") pod \"dede6da5-7e4f-40aa-9601-08827f72f516\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") "
Sep 18 18:23:02.976765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-model-cache\") pod \"dede6da5-7e4f-40aa-9601-08827f72f516\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") "
Sep 18 18:23:02.976973 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-dshm\") pod \"dede6da5-7e4f-40aa-9601-08827f72f516\" (UID: \"dede6da5-7e4f-40aa-9601-08827f72f516\") "
Sep 18 18:23:02.976973 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-home" (OuterVolumeSpecName: "home") pod "dede6da5-7e4f-40aa-9601-08827f72f516" (UID: "dede6da5-7e4f-40aa-9601-08827f72f516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:23:02.976973 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.976930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dede6da5-7e4f-40aa-9601-08827f72f516" (UID: "dede6da5-7e4f-40aa-9601-08827f72f516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:23:02.977164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.977008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-model-cache" (OuterVolumeSpecName: "model-cache") pod "dede6da5-7e4f-40aa-9601-08827f72f516" (UID: "dede6da5-7e4f-40aa-9601-08827f72f516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:23:02.977164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.977142    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:23:02.977164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.977159    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:23:02.977295 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.977172    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:23:02.978898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.978867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dede6da5-7e4f-40aa-9601-08827f72f516-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dede6da5-7e4f-40aa-9601-08827f72f516" (UID: "dede6da5-7e4f-40aa-9601-08827f72f516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:23:02.979005 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.978915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-dshm" (OuterVolumeSpecName: "dshm") pod "dede6da5-7e4f-40aa-9601-08827f72f516" (UID: "dede6da5-7e4f-40aa-9601-08827f72f516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:23:02.979005 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:02.978929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dede6da5-7e4f-40aa-9601-08827f72f516-kube-api-access-zx9mq" (OuterVolumeSpecName: "kube-api-access-zx9mq") pod "dede6da5-7e4f-40aa-9601-08827f72f516" (UID: "dede6da5-7e4f-40aa-9601-08827f72f516"). InnerVolumeSpecName "kube-api-access-zx9mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:23:03.078254 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.078217    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dede6da5-7e4f-40aa-9601-08827f72f516-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:23:03.078405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.078258    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9mq\" (UniqueName: \"kubernetes.io/projected/dede6da5-7e4f-40aa-9601-08827f72f516-kube-api-access-zx9mq\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:23:03.078405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.078275    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dede6da5-7e4f-40aa-9601-08827f72f516-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:23:03.362070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.361966    2580 generic.go:356] "Generic (PLEG): container finished" podID="dede6da5-7e4f-40aa-9601-08827f72f516" containerID="e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183" exitCode=0
Sep 18 18:23:03.362070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.362011    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"
Sep 18 18:23:03.362526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.362055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn" event={"ID":"dede6da5-7e4f-40aa-9601-08827f72f516","Type":"ContainerDied","Data":"e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183"}
Sep 18 18:23:03.362526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.362116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn" event={"ID":"dede6da5-7e4f-40aa-9601-08827f72f516","Type":"ContainerDied","Data":"c6f69473cc412591f717395b7447c40b1e78cb3d05835d7443e3c3c9d16365c0"}
Sep 18 18:23:03.362526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.362133    2580 scope.go:117] "RemoveContainer" containerID="e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183"
Sep 18 18:23:03.371197 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.371177    2580 scope.go:117] "RemoveContainer" containerID="e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183"
Sep 18 18:23:03.371475 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:23:03.371455    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183\": container with ID starting with e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183 not found: ID does not exist" containerID="e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183"
Sep 18 18:23:03.371520 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.371484    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183"} err="failed to get container status \"e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183\": rpc error: code = NotFound desc = could not find container \"e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183\": container with ID starting with e9e8763d1fc7fbcf04313c331c23455702b4389b6fb96ae030e8abdd28593183 not found: ID does not exist"
Sep 18 18:23:03.383633 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.383606    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"]
Sep 18 18:23:03.388641 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.388621    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6fa1a2a2/e2e-lifecycle-model-6fa1a2a2-kserve-86557c75b7-zqnbn"]
Sep 18 18:23:03.910729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:03.910687    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dede6da5-7e4f-40aa-9601-08827f72f516" path="/var/lib/kubelet/pods/dede6da5-7e4f-40aa-9601-08827f72f516/volumes"
Sep 18 18:23:22.900968 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:22.900905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:24.120617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:24.120588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:24.645819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:24.645785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:30.092479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:30.092441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:35.791473 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:35.791431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:35.891157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:35.891116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:36.019324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:36.019271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:37.225540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:37.225499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:37.323576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:37.323514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:37.447843 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:37.447809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:38.013521 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:38.013474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:38.111741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:38.111704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:38.213676 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:38.213629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:39.467708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:39.467653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:45.482999 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:45.482958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:45.583132 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:45.583097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:46.163638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:46.163599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:51.982753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:51.982718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:52.048864 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:52.048830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:52.134546 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:52.134502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:52.234609 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:52.234461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:58.709492 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:58.709460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:59.147693 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:59.147602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:59.801822 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:23:59.801785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:00.635419 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:00.635377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:01.256419 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:01.256379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:02.008218 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:02.008190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:02.645869 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:02.645833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:02.819466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:02.819427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:02.920695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:02.920661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:03.403769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:03.403681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:03.706404 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:03.706372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"]
Sep 18 18:24:03.795015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:03.794982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:13.564231 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:13.564152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:14.447301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.447264    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"]
Sep 18 18:24:14.447811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.447778    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k" podUID="87b3bce8-e817-4fa2-8b60-584ae6a4508e" containerName="maas-api" containerID="cri-o://e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a" gracePeriod=30
Sep 18 18:24:14.479944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.479903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:14.580190 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.580148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:14.679474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.679432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:14.693044 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.693010    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:24:14.733018 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.732947    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b3bce8-e817-4fa2-8b60-584ae6a4508e" containerID="e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a" exitCode=0
Sep 18 18:24:14.733018 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.733006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"
Sep 18 18:24:14.733201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.733063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k" event={"ID":"87b3bce8-e817-4fa2-8b60-584ae6a4508e","Type":"ContainerDied","Data":"e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a"}
Sep 18 18:24:14.733201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.733128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k" event={"ID":"87b3bce8-e817-4fa2-8b60-584ae6a4508e","Type":"ContainerDied","Data":"8445b240b81ab00c11d205222349fe6df7c91b59c2e86543eb567594314438c5"}
Sep 18 18:24:14.733201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.733162    2580 scope.go:117] "RemoveContainer" containerID="e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a"
Sep 18 18:24:14.743345 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.743326    2580 scope.go:117] "RemoveContainer" containerID="e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a"
Sep 18 18:24:14.743619 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:24:14.743599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a\": container with ID starting with e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a not found: ID does not exist" containerID="e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a"
Sep 18 18:24:14.743703 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.743632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a"} err="failed to get container status \"e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a\": rpc error: code = NotFound desc = could not find container \"e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a\": container with ID starting with e827d91dda34a8cb058aa9f117b9ca42d114d1a0d954af4c91af3174b6774f2a not found: ID does not exist"
Sep 18 18:24:14.746462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.746431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:14.829723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.829698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-maas-api-tls\") pod \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") "
Sep 18 18:24:14.829861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.829791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b3bce8-e817-4fa2-8b60-584ae6a4508e-openshift-service-ca\") pod \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") "
Sep 18 18:24:14.829861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.829824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-metrics-certs\") pod \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") "
Sep 18 18:24:14.829981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.829865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6brp\" (UniqueName: \"kubernetes.io/projected/87b3bce8-e817-4fa2-8b60-584ae6a4508e-kube-api-access-h6brp\") pod \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\" (UID: \"87b3bce8-e817-4fa2-8b60-584ae6a4508e\") "
Sep 18 18:24:14.830462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.830425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b3bce8-e817-4fa2-8b60-584ae6a4508e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87b3bce8-e817-4fa2-8b60-584ae6a4508e" (UID: "87b3bce8-e817-4fa2-8b60-584ae6a4508e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:24:14.833695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.833245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b3bce8-e817-4fa2-8b60-584ae6a4508e-kube-api-access-h6brp" (OuterVolumeSpecName: "kube-api-access-h6brp") pod "87b3bce8-e817-4fa2-8b60-584ae6a4508e" (UID: "87b3bce8-e817-4fa2-8b60-584ae6a4508e"). InnerVolumeSpecName "kube-api-access-h6brp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:24:14.835240 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.835218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87b3bce8-e817-4fa2-8b60-584ae6a4508e" (UID: "87b3bce8-e817-4fa2-8b60-584ae6a4508e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:24:14.835322 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.835235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87b3bce8-e817-4fa2-8b60-584ae6a4508e" (UID: "87b3bce8-e817-4fa2-8b60-584ae6a4508e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:24:14.931232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.931181    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:14.931232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.931222    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6brp\" (UniqueName: \"kubernetes.io/projected/87b3bce8-e817-4fa2-8b60-584ae6a4508e-kube-api-access-h6brp\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:14.931232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.931233    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b3bce8-e817-4fa2-8b60-584ae6a4508e-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:14.931232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:14.931241    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b3bce8-e817-4fa2-8b60-584ae6a4508e-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:15.050300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:15.050265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:15.054962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:15.054939    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"]
Sep 18 18:24:15.060912 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:15.060891    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-37c153-b6f8cb64d-4gm6k"]
Sep 18 18:24:15.215239 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:15.215207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:15.905800 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:15.905744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:15.910753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:15.910727    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b3bce8-e817-4fa2-8b60-584ae6a4508e" path="/var/lib/kubelet/pods/87b3bce8-e817-4fa2-8b60-584ae6a4508e/volumes"
Sep 18 18:24:16.813659 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:16.813622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:17.615835 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:17.615792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:18.214064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:18.214007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:18.726265 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:18.726226    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz" podUID="3513d1c4-6eb3-4013-a53a-a5408b1a2e62" containerName="main" containerID="cri-o://0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34" gracePeriod=45
Sep 18 18:24:18.978347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:18.978289    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:24:19.066377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-model-cache\") pod \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") "
Sep 18 18:24:19.066555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tmp-dir\") pod \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") "
Sep 18 18:24:19.066555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsgs9\" (UniqueName: \"kubernetes.io/projected/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-kube-api-access-gsgs9\") pod \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") "
Sep 18 18:24:19.066555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tls-certs\") pod \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") "
Sep 18 18:24:19.066555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-dshm\") pod \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") "
Sep 18 18:24:19.066555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-home\") pod \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\" (UID: \"3513d1c4-6eb3-4013-a53a-a5408b1a2e62\") "
Sep 18 18:24:19.066783 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-model-cache" (OuterVolumeSpecName: "model-cache") pod "3513d1c4-6eb3-4013-a53a-a5408b1a2e62" (UID: "3513d1c4-6eb3-4013-a53a-a5408b1a2e62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:24:19.066783 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3513d1c4-6eb3-4013-a53a-a5408b1a2e62" (UID: "3513d1c4-6eb3-4013-a53a-a5408b1a2e62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:24:19.066889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-home" (OuterVolumeSpecName: "home") pod "3513d1c4-6eb3-4013-a53a-a5408b1a2e62" (UID: "3513d1c4-6eb3-4013-a53a-a5408b1a2e62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:24:19.066889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066845    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:19.066889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.066860    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:19.069131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.069075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-dshm" (OuterVolumeSpecName: "dshm") pod "3513d1c4-6eb3-4013-a53a-a5408b1a2e62" (UID: "3513d1c4-6eb3-4013-a53a-a5408b1a2e62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:24:19.069131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.069096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3513d1c4-6eb3-4013-a53a-a5408b1a2e62" (UID: "3513d1c4-6eb3-4013-a53a-a5408b1a2e62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:24:19.069131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.069106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-kube-api-access-gsgs9" (OuterVolumeSpecName: "kube-api-access-gsgs9") pod "3513d1c4-6eb3-4013-a53a-a5408b1a2e62" (UID: "3513d1c4-6eb3-4013-a53a-a5408b1a2e62"). InnerVolumeSpecName "kube-api-access-gsgs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:24:19.168054 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.168005    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgs9\" (UniqueName: \"kubernetes.io/projected/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-kube-api-access-gsgs9\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:19.168205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.168056    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:19.168205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.168074    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:19.168205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.168086    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3513d1c4-6eb3-4013-a53a-a5408b1a2e62-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:24:19.760684 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.760651    2580 generic.go:356] "Generic (PLEG): container finished" podID="3513d1c4-6eb3-4013-a53a-a5408b1a2e62" containerID="0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34" exitCode=0
Sep 18 18:24:19.761206 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.760695    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"
Sep 18 18:24:19.761206 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.760730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz" event={"ID":"3513d1c4-6eb3-4013-a53a-a5408b1a2e62","Type":"ContainerDied","Data":"0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34"}
Sep 18 18:24:19.761206 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.760771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz" event={"ID":"3513d1c4-6eb3-4013-a53a-a5408b1a2e62","Type":"ContainerDied","Data":"29bf0be4c44a438319e435fd0da85e3ab31c2da5d8a19e89885b26d0b1d4ef0d"}
Sep 18 18:24:19.761206 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.760795    2580 scope.go:117] "RemoveContainer" containerID="0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34"
Sep 18 18:24:19.771587 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.771566    2580 scope.go:117] "RemoveContainer" containerID="0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34"
Sep 18 18:24:19.772261 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:24:19.772229    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34\": container with ID starting with 0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34 not found: ID does not exist" containerID="0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34"
Sep 18 18:24:19.772386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.772275    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34"} err="failed to get container status \"0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34\": rpc error: code = NotFound desc = could not find container \"0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34\": container with ID starting with 0aebc2512f5aaf32801d704c8b030bb11a5eb52b615e4e4f4aa965fd25ca1e34 not found: ID does not exist"
Sep 18 18:24:19.782729 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.782708    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"]
Sep 18 18:24:19.787566 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.787547    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-37c153/distinct-w0-37c153-kserve-5fddb5c8b9-pvcwz"]
Sep 18 18:24:19.909975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:19.909944    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3513d1c4-6eb3-4013-a53a-a5408b1a2e62" path="/var/lib/kubelet/pods/3513d1c4-6eb3-4013-a53a-a5408b1a2e62/volumes"
Sep 18 18:24:21.011320 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:21.011289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:21.112389 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:21.112348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:21.299129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:21.299008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:23.810915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:23.810866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:24.618459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:24.618422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:29.888061 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:29.887999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:29.987709 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:29.987666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:30.217127 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:30.217098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:30.526050 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:30.525934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:30.642513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:30.642480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:33.231264 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:33.231227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:34.036669 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:34.036636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:34.138114 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:34.138082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:34.901841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:34.901807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:35.751327 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:35.751290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:36.455093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:36.455051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:37.194183 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194147    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"]
Sep 18 18:24:37.194769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194737    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a2da44-97d4-47fa-b6ee-e7763deb8617" containerName="maas-api"
Sep 18 18:24:37.194769 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194765    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a2da44-97d4-47fa-b6ee-e7763deb8617" containerName="maas-api"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194788    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194799    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3513d1c4-6eb3-4013-a53a-a5408b1a2e62" containerName="main"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194808    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3513d1c4-6eb3-4013-a53a-a5408b1a2e62" containerName="main"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b3bce8-e817-4fa2-8b60-584ae6a4508e" containerName="maas-api"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194830    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b3bce8-e817-4fa2-8b60-584ae6a4508e" containerName="maas-api"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194844    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b2ebbb-8ddc-4e24-ac16-79655fefdecb" containerName="main"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194852    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b2ebbb-8ddc-4e24-ac16-79655fefdecb" containerName="main"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c1feb3-d737-40b3-9e19-85d841df3965" containerName="payload-processing"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194883    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c1feb3-d737-40b3-9e19-85d841df3965" containerName="payload-processing"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194896    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dede6da5-7e4f-40aa-9601-08827f72f516" containerName="main"
Sep 18 18:24:37.194963 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.194904    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dede6da5-7e4f-40aa-9601-08827f72f516" containerName="main"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195050    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a2da44-97d4-47fa-b6ee-e7763deb8617" containerName="maas-api"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195067    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3513d1c4-6eb3-4013-a53a-a5408b1a2e62" containerName="main"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195077    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195089    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195140    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c1feb3-d737-40b3-9e19-85d841df3965" containerName="payload-processing"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195151    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b3bce8-e817-4fa2-8b60-584ae6a4508e" containerName="maas-api"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dede6da5-7e4f-40aa-9601-08827f72f516" containerName="main"
Sep 18 18:24:37.195615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.195169    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b2ebbb-8ddc-4e24-ac16-79655fefdecb" containerName="main"
Sep 18 18:24:37.200782 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.200761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.207051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.207010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e6108f-serving-cert\""
Sep 18 18:24:37.209988 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.209129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"]
Sep 18 18:24:37.226424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.226395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:37.283411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.283383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:37.331021 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.330987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e817a2-893c-4802-8304-ec3e5ab690ca-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.331208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.331058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-maas-api-tls\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.331208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.331164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-metrics-certs\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.331208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.331199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/90e817a2-893c-4802-8304-ec3e5ab690ca-kube-api-access-p659n\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.386324 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.386293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:37.432393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.432364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-metrics-certs\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.432517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.432400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/90e817a2-893c-4802-8304-ec3e5ab690ca-kube-api-access-p659n\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.432517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.432449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e817a2-893c-4802-8304-ec3e5ab690ca-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.432517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.432474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-maas-api-tls\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.433266 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.433204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e817a2-893c-4802-8304-ec3e5ab690ca-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.435069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.435042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-maas-api-tls\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.435168 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.435105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-metrics-certs\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.440282 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.440263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/90e817a2-893c-4802-8304-ec3e5ab690ca-kube-api-access-p659n\") pod \"maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.514558 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.514502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:37.755053 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.755002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:37.786994 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.786922    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"]
Sep 18 18:24:37.787568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.787545    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy"
Sep 18 18:24:37.787568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.787569    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ec0aca-3d3a-4a25-9d71-e80034f52d0a" containerName="istio-proxy"
Sep 18 18:24:37.791132 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.791115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"]
Sep 18 18:24:37.791217 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.791206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:37.792934 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:24:37.792914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e817a2_893c_4802_8304_ec3e5ab690ca.slice/crio-31691d2a51cdf8214e9f467231019539ef2357e0e189bfaba82f6ba1eca1da25 WatchSource:0}: Error finding container 31691d2a51cdf8214e9f467231019539ef2357e0e189bfaba82f6ba1eca1da25: Status 404 returned error can't find the container with id 31691d2a51cdf8214e9f467231019539ef2357e0e189bfaba82f6ba1eca1da25
Sep 18 18:24:37.793652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.793620    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e6108f\""
Sep 18 18:24:37.793767 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.793734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e6108f-dockercfg-ghkln\""
Sep 18 18:24:37.794522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.794504    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:24:37.804369 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.802209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"]
Sep 18 18:24:37.830893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.830864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:37.851783 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.851745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4" event={"ID":"90e817a2-893c-4802-8304-ec3e5ab690ca","Type":"ContainerStarted","Data":"31691d2a51cdf8214e9f467231019539ef2357e0e189bfaba82f6ba1eca1da25"}
Sep 18 18:24:37.936778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.936741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27953c2f-224c-437b-bb29-62c6126df108-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:37.936966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:37.936791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4h7\" (UniqueName: \"kubernetes.io/projected/27953c2f-224c-437b-bb29-62c6126df108-kube-api-access-4d4h7\") pod \"payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.038088 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.038001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27953c2f-224c-437b-bb29-62c6126df108-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.038088 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.038080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4h7\" (UniqueName: \"kubernetes.io/projected/27953c2f-224c-437b-bb29-62c6126df108-kube-api-access-4d4h7\") pod \"payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.038759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.038738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27953c2f-224c-437b-bb29-62c6126df108-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.045466 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.045447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4h7\" (UniqueName: \"kubernetes.io/projected/27953c2f-224c-437b-bb29-62c6126df108-kube-api-access-4d4h7\") pod \"payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.120062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.120019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.393111 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.393082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"]
Sep 18 18:24:38.394722 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:24:38.394687    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27953c2f_224c_437b_bb29_62c6126df108.slice/crio-9f036a5cee30af9518b9b9c82435754f9cfb2909e060112776ff86bf0896559e WatchSource:0}: Error finding container 9f036a5cee30af9518b9b9c82435754f9cfb2909e060112776ff86bf0896559e: Status 404 returned error can't find the container with id 9f036a5cee30af9518b9b9c82435754f9cfb2909e060112776ff86bf0896559e
Sep 18 18:24:38.859539 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.859489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7" event={"ID":"27953c2f-224c-437b-bb29-62c6126df108","Type":"ContainerStarted","Data":"7fa9d820796a40872239d0aad9fa0bde559de721e9e80bf89ed72f9a04b49a93"}
Sep 18 18:24:38.860014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.859565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7" event={"ID":"27953c2f-224c-437b-bb29-62c6126df108","Type":"ContainerStarted","Data":"9f036a5cee30af9518b9b9c82435754f9cfb2909e060112776ff86bf0896559e"}
Sep 18 18:24:38.860014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.859596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:38.861337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.861309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4" event={"ID":"90e817a2-893c-4802-8304-ec3e5ab690ca","Type":"ContainerStarted","Data":"08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c"}
Sep 18 18:24:38.861459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.861373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:38.875907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.875862    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7" podStartSLOduration=1.875850147 podStartE2EDuration="1.875850147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:24:38.874103346 +0000 UTC m=+1449.025978181" watchObservedRunningTime="2026-09-18 18:24:38.875850147 +0000 UTC m=+1449.027724981"
Sep 18 18:24:38.889748 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:38.889703    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4" podStartSLOduration=1.506246715 podStartE2EDuration="1.889689209s" totalImagesPullingTime="383.442494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:24:38.888301168 +0000 UTC m=+1449.040176003" watchObservedRunningTime="2026-09-18 18:24:38.889689209 +0000 UTC m=+1449.041564044"
Sep 18 18:24:41.634476 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:41.634436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:41.734717 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:41.734675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:43.533689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:43.533626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:44.872988 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:44.872953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:24:46.807445 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:46.807416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:49.870083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:49.870053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:24:56.471366 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:56.471331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:57.849197 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:57.849157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:58.093615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:58.093581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:59.385001 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:24:59.384971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:00.167898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:00.167852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:00.849078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:00.849020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:01.593396 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:01.593361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:02.140824 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:02.140789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:02.809611 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:02.809575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:04.108738 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.108702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:04.568098 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.568015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:04.831488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.831404    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"]
Sep 18 18:25:04.835165 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.835141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.837642 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.837624    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-29f0cb-openshift-default-dockercfg-2prw2\""
Sep 18 18:25:04.853582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.853556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"]
Sep 18 18:25:04.862240 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.862198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:04.975846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.975805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-certs\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.975846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.975846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-socket\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.975868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f378c09a-0272-4a54-b476-02721bc3272a-istiod-ca-cert\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.975904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-envoy\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.975931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f378c09a-0272-4a54-b476-02721bc3272a-istio-podinfo\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.976010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-data\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.976101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-credential-socket\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.976165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zds45\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-kube-api-access-zds45\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:04.976235 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:04.976191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-istio-token\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077404 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-istio-token\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-certs\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-socket\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f378c09a-0272-4a54-b476-02721bc3272a-istiod-ca-cert\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-envoy\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f378c09a-0272-4a54-b476-02721bc3272a-istio-podinfo\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-data\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-credential-socket\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.077834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zds45\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-kube-api-access-zds45\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.078004 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-certs\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.078004 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.077862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-socket\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.078162 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.078101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-data\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.078225 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.078214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-credential-socket\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.078352 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.078329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f378c09a-0272-4a54-b476-02721bc3272a-istiod-ca-cert\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.079990 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.079968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-envoy\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.080232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.080215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f378c09a-0272-4a54-b476-02721bc3272a-istio-podinfo\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.083332 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.083268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:05.085334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.085310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zds45\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-kube-api-access-zds45\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.085334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.085322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-istio-token\") pod \"e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") " pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.146087 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.146055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:05.420226 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.420201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"]
Sep 18 18:25:05.422258 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:25:05.422226    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf378c09a_0272_4a54_b476_02721bc3272a.slice/crio-ec66ade5a038675f18063065382846e4362c5eb046ff07622f193d768f29132e WatchSource:0}: Error finding container ec66ade5a038675f18063065382846e4362c5eb046ff07622f193d768f29132e: Status 404 returned error can't find the container with id ec66ade5a038675f18063065382846e4362c5eb046ff07622f193d768f29132e
Sep 18 18:25:05.424825 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.424781    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:25:05.424902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.424875    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:25:05.424981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:05.424918    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:25:06.011635 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:06.011600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" event={"ID":"f378c09a-0272-4a54-b476-02721bc3272a","Type":"ContainerStarted","Data":"85781297e59a37aa06f58a9451f2ca4340bce2212d9d2731cba5a360f89587fc"}
Sep 18 18:25:06.011813 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:06.011671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" event={"ID":"f378c09a-0272-4a54-b476-02721bc3272a","Type":"ContainerStarted","Data":"ec66ade5a038675f18063065382846e4362c5eb046ff07622f193d768f29132e"}
Sep 18 18:25:06.033785 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:06.033726    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podStartSLOduration=2.033711635 podStartE2EDuration="2.033711635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:25:06.031492886 +0000 UTC m=+1476.183367750" watchObservedRunningTime="2026-09-18 18:25:06.033711635 +0000 UTC m=+1476.185586470"
Sep 18 18:25:06.146957 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:06.146915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:06.153919 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:06.153892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:07.016995 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:07.016970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:07.018093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:07.018070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:07.313491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:07.313404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:07.980548 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:07.980505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:08.067634 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:08.067596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:08.166945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:08.166914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:09.021006 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:09.020974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:09.697667 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:09.697636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:10.521596 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:10.521539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:11.151996 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:11.151959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:11.826186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:11.826149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:11.890335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:11.890303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:12.465548 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.465509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:12.557631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.557589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:12.561694 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.561669    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"]
Sep 18 18:25:12.566605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.566588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.568928 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.568910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-29f0cb\""
Sep 18 18:25:12.569108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.569094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-29f0cb-dockercfg-x7xpr\""
Sep 18 18:25:12.578244 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.578225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"]
Sep 18 18:25:12.655654 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.655610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934bf613-d52b-4c98-83df-bbe022881572-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.655868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.655665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8r8\" (UniqueName: \"kubernetes.io/projected/934bf613-d52b-4c98-83df-bbe022881572-kube-api-access-tk8r8\") pod \"payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.757085 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.756985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934bf613-d52b-4c98-83df-bbe022881572-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.757085 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.757020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8r8\" (UniqueName: \"kubernetes.io/projected/934bf613-d52b-4c98-83df-bbe022881572-kube-api-access-tk8r8\") pod \"payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.757742 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.757719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934bf613-d52b-4c98-83df-bbe022881572-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.764520 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.764495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8r8\" (UniqueName: \"kubernetes.io/projected/934bf613-d52b-4c98-83df-bbe022881572-kube-api-access-tk8r8\") pod \"payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:12.877517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:12.877491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:13.150322 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:13.150298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"]
Sep 18 18:25:13.152076 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:25:13.152003    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934bf613_d52b_4c98_83df_bbe022881572.slice/crio-375f97e186b4bc71d8b560c4201895d264ed559c5375f9d711d32f0e71ba107b WatchSource:0}: Error finding container 375f97e186b4bc71d8b560c4201895d264ed559c5375f9d711d32f0e71ba107b: Status 404 returned error can't find the container with id 375f97e186b4bc71d8b560c4201895d264ed559c5375f9d711d32f0e71ba107b
Sep 18 18:25:14.053617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:14.053580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld" event={"ID":"934bf613-d52b-4c98-83df-bbe022881572","Type":"ContainerStarted","Data":"11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a"}
Sep 18 18:25:14.054095 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:14.053639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld" event={"ID":"934bf613-d52b-4c98-83df-bbe022881572","Type":"ContainerStarted","Data":"375f97e186b4bc71d8b560c4201895d264ed559c5375f9d711d32f0e71ba107b"}
Sep 18 18:25:14.054095 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:14.053671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:14.070905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:14.070860    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld" podStartSLOduration=2.07084432 podStartE2EDuration="2.07084432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:25:14.070189118 +0000 UTC m=+1484.222063957" watchObservedRunningTime="2026-09-18 18:25:14.07084432 +0000 UTC m=+1484.222719155"
Sep 18 18:25:17.176272 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:17.176235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:18.182803 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:18.182773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:20.601583 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:20.601549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:23.810604 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:23.810563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:25.060947 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:25.060919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:25:33.155490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.155454    2580 generic.go:356] "Generic (PLEG): container finished" podID="f378c09a-0272-4a54-b476-02721bc3272a" containerID="85781297e59a37aa06f58a9451f2ca4340bce2212d9d2731cba5a360f89587fc" exitCode=0
Sep 18 18:25:33.156007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.155534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" event={"ID":"f378c09a-0272-4a54-b476-02721bc3272a","Type":"ContainerDied","Data":"85781297e59a37aa06f58a9451f2ca4340bce2212d9d2731cba5a360f89587fc"}
Sep 18 18:25:33.156106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.156079    2580 scope.go:117] "RemoveContainer" containerID="85781297e59a37aa06f58a9451f2ca4340bce2212d9d2731cba5a360f89587fc"
Sep 18 18:25:33.157190 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.157152    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:25:33.157284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.157224    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:25:33.157284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.157277    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:25:33.734349 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:33.734315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:34.165536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:34.165492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" event={"ID":"f378c09a-0272-4a54-b476-02721bc3272a","Type":"ContainerStarted","Data":"9ff958d8ab1984e6cf3fc1c1903d83d6cfbe8a49687207aecb8d60632c758eff"}
Sep 18 18:25:35.060776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:35.060742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:35.147620 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:35.147577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:35.147826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:35.147667    2580 patch_prober.go:28] interesting pod/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 18 18:25:35.147826 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:35.147727    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 18 18:25:35.852986 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:35.852953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:36.147424 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:36.147334    2580 patch_prober.go:28] interesting pod/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 18 18:25:36.147703 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:36.147673    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 18 18:25:36.504679 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:36.504644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:37.017203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:37.017173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:37.147177 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:37.147131    2580 patch_prober.go:28] interesting pod/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 18 18:25:37.147370 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:37.147222    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 18 18:25:37.202983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:37.202941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:38.017658 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:38.017621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:38.118637 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:38.118605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:38.147517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:38.147091    2580 patch_prober.go:28] interesting pod/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 18 18:25:38.147517 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:38.147162    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 18 18:25:38.764695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:38.764658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:39.147906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:39.147404    2580 patch_prober.go:28] interesting pod/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 18 18:25:39.147906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:39.147474    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 18 18:25:39.501898 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:39.501855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:40.150902 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:40.150871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:40.151927 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:40.151908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:25:40.598708 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:40.598665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:41.513363 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:41.513335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:42.193360 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:42.193330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:42.899317 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:42.899279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:43.551114 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:43.551077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:44.131452 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:44.131413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:44.220870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:44.220831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:44.976626 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:44.976589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:45.739192 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:45.739151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:48.214311 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:48.214277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:48.269188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:48.269151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:51.264551 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:51.264515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:56.933692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:56.933480    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"]
Sep 18 18:25:56.934759 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:56.934728    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4" podUID="90e817a2-893c-4802-8304-ec3e5ab690ca" containerName="maas-api" containerID="cri-o://08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c" gracePeriod=30
Sep 18 18:25:56.962544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:56.962504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.002347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.002315    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"]
Sep 18 18:25:57.002617 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.002587    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7" podUID="27953c2f-224c-437b-bb29-62c6126df108" containerName="payload-pre-processing" containerID="cri-o://7fa9d820796a40872239d0aad9fa0bde559de721e9e80bf89ed72f9a04b49a93" gracePeriod=30
Sep 18 18:25:57.061145 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.061002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.161756 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.161718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.269378 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.269335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.272079 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.272054    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:25:57.295547 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.295217    2580 generic.go:356] "Generic (PLEG): container finished" podID="90e817a2-893c-4802-8304-ec3e5ab690ca" containerID="08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c" exitCode=0
Sep 18 18:25:57.295547 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.295266    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"
Sep 18 18:25:57.295547 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.295352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4" event={"ID":"90e817a2-893c-4802-8304-ec3e5ab690ca","Type":"ContainerDied","Data":"08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c"}
Sep 18 18:25:57.295547 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.295401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4" event={"ID":"90e817a2-893c-4802-8304-ec3e5ab690ca","Type":"ContainerDied","Data":"31691d2a51cdf8214e9f467231019539ef2357e0e189bfaba82f6ba1eca1da25"}
Sep 18 18:25:57.295547 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.295434    2580 scope.go:117] "RemoveContainer" containerID="08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c"
Sep 18 18:25:57.297497 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.297400    2580 generic.go:356] "Generic (PLEG): container finished" podID="27953c2f-224c-437b-bb29-62c6126df108" containerID="7fa9d820796a40872239d0aad9fa0bde559de721e9e80bf89ed72f9a04b49a93" exitCode=0
Sep 18 18:25:57.297497 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.297442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7" event={"ID":"27953c2f-224c-437b-bb29-62c6126df108","Type":"ContainerDied","Data":"7fa9d820796a40872239d0aad9fa0bde559de721e9e80bf89ed72f9a04b49a93"}
Sep 18 18:25:57.297497 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.297496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7" event={"ID":"27953c2f-224c-437b-bb29-62c6126df108","Type":"ContainerDied","Data":"9f036a5cee30af9518b9b9c82435754f9cfb2909e060112776ff86bf0896559e"}
Sep 18 18:25:57.297707 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.297519    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f036a5cee30af9518b9b9c82435754f9cfb2909e060112776ff86bf0896559e"
Sep 18 18:25:57.302402 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.302385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:25:57.305908 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.305892    2580 scope.go:117] "RemoveContainer" containerID="08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c"
Sep 18 18:25:57.306227 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:25:57.306210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c\": container with ID starting with 08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c not found: ID does not exist" containerID="08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c"
Sep 18 18:25:57.306294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.306234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c"} err="failed to get container status \"08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c\": rpc error: code = NotFound desc = could not find container \"08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c\": container with ID starting with 08cb44612fdf5f8ae903d3b552afeeb01256e12b8bb44035b26de1c17ad4f33c not found: ID does not exist"
Sep 18 18:25:57.400488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.400456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/90e817a2-893c-4802-8304-ec3e5ab690ca-kube-api-access-p659n\") pod \"90e817a2-893c-4802-8304-ec3e5ab690ca\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") "
Sep 18 18:25:57.400676 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.400662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e817a2-893c-4802-8304-ec3e5ab690ca-openshift-service-ca\") pod \"90e817a2-893c-4802-8304-ec3e5ab690ca\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") "
Sep 18 18:25:57.400725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.400691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d4h7\" (UniqueName: \"kubernetes.io/projected/27953c2f-224c-437b-bb29-62c6126df108-kube-api-access-4d4h7\") pod \"27953c2f-224c-437b-bb29-62c6126df108\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") "
Sep 18 18:25:57.400725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.400714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27953c2f-224c-437b-bb29-62c6126df108-plugins-config-volume\") pod \"27953c2f-224c-437b-bb29-62c6126df108\" (UID: \"27953c2f-224c-437b-bb29-62c6126df108\") "
Sep 18 18:25:57.400811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.400753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-metrics-certs\") pod \"90e817a2-893c-4802-8304-ec3e5ab690ca\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") "
Sep 18 18:25:57.400811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.400782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-maas-api-tls\") pod \"90e817a2-893c-4802-8304-ec3e5ab690ca\" (UID: \"90e817a2-893c-4802-8304-ec3e5ab690ca\") "
Sep 18 18:25:57.401093 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.401065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e817a2-893c-4802-8304-ec3e5ab690ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90e817a2-893c-4802-8304-ec3e5ab690ca" (UID: "90e817a2-893c-4802-8304-ec3e5ab690ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:25:57.401201 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.401114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27953c2f-224c-437b-bb29-62c6126df108-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27953c2f-224c-437b-bb29-62c6126df108" (UID: "27953c2f-224c-437b-bb29-62c6126df108"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:25:57.402906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.402884    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90e817a2-893c-4802-8304-ec3e5ab690ca" (UID: "90e817a2-893c-4802-8304-ec3e5ab690ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:25:57.403051 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.403006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90e817a2-893c-4802-8304-ec3e5ab690ca" (UID: "90e817a2-893c-4802-8304-ec3e5ab690ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:25:57.403373 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.403355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27953c2f-224c-437b-bb29-62c6126df108-kube-api-access-4d4h7" (OuterVolumeSpecName: "kube-api-access-4d4h7") pod "27953c2f-224c-437b-bb29-62c6126df108" (UID: "27953c2f-224c-437b-bb29-62c6126df108"). InnerVolumeSpecName "kube-api-access-4d4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:25:57.403431 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.403371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e817a2-893c-4802-8304-ec3e5ab690ca-kube-api-access-p659n" (OuterVolumeSpecName: "kube-api-access-p659n") pod "90e817a2-893c-4802-8304-ec3e5ab690ca" (UID: "90e817a2-893c-4802-8304-ec3e5ab690ca"). InnerVolumeSpecName "kube-api-access-p659n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:25:57.502149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.502095    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e817a2-893c-4802-8304-ec3e5ab690ca-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:25:57.502149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.502117    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4h7\" (UniqueName: \"kubernetes.io/projected/27953c2f-224c-437b-bb29-62c6126df108-kube-api-access-4d4h7\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:25:57.502149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.502126    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27953c2f-224c-437b-bb29-62c6126df108-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:25:57.502149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.502134    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:25:57.502149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.502142    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e817a2-893c-4802-8304-ec3e5ab690ca-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:25:57.502149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.502149    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/90e817a2-893c-4802-8304-ec3e5ab690ca-kube-api-access-p659n\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:25:57.542353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.542314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.618012 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.617977    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"]
Sep 18 18:25:57.621354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.621329    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6108f-67bdb658b6-s7qm4"]
Sep 18 18:25:57.624223 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.624197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.911354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:57.911326    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90e817a2-893c-4802-8304-ec3e5ab690ca" path="/var/lib/kubelet/pods/90e817a2-893c-4802-8304-ec3e5ab690ca/volumes"
Sep 18 18:25:58.299749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:58.299665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:58.304597 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:58.304576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"
Sep 18 18:25:58.319276 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:58.319250    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"]
Sep 18 18:25:58.324944 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:58.324918    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6108f-67cc9f8dd6-2bhw7"]
Sep 18 18:25:58.942577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:58.942541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:59.728304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:59.728262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:59.910310 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:25:59.910272    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27953c2f-224c-437b-bb29-62c6126df108" path="/var/lib/kubelet/pods/27953c2f-224c-437b-bb29-62c6126df108/volumes"
Sep 18 18:26:00.381948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:00.381911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:01.008549 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:01.008513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:03.621961 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:03.621926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:03.731668 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:03.731612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:03.867676 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:03.867640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:04.390285 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:04.390225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.301678 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:05.301638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.992202 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:05.992159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:06.670381 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:06.670346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:07.335819 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:07.335778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:08.029510 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:08.029470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:08.984793 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:08.984756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:11.615416 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:11.615357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:14.513559 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:14.513522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:20.304883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.304842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:20.343888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.343857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"]
Sep 18 18:26:20.344113 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.344089    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld" podUID="934bf613-d52b-4c98-83df-bbe022881572" containerName="payload-pre-processing" containerID="cri-o://11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a" gracePeriod=30
Sep 18 18:26:20.404741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.404708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:20.505122 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.505083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:20.597258 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.597236    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:26:20.606354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.606322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:20.717292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.717264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk8r8\" (UniqueName: \"kubernetes.io/projected/934bf613-d52b-4c98-83df-bbe022881572-kube-api-access-tk8r8\") pod \"934bf613-d52b-4c98-83df-bbe022881572\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") "
Sep 18 18:26:20.717479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.717313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934bf613-d52b-4c98-83df-bbe022881572-plugins-config-volume\") pod \"934bf613-d52b-4c98-83df-bbe022881572\" (UID: \"934bf613-d52b-4c98-83df-bbe022881572\") "
Sep 18 18:26:20.717660 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.717633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934bf613-d52b-4c98-83df-bbe022881572-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "934bf613-d52b-4c98-83df-bbe022881572" (UID: "934bf613-d52b-4c98-83df-bbe022881572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:20.719491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.719473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934bf613-d52b-4c98-83df-bbe022881572-kube-api-access-tk8r8" (OuterVolumeSpecName: "kube-api-access-tk8r8") pod "934bf613-d52b-4c98-83df-bbe022881572" (UID: "934bf613-d52b-4c98-83df-bbe022881572"). InnerVolumeSpecName "kube-api-access-tk8r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:20.818885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.818852    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk8r8\" (UniqueName: \"kubernetes.io/projected/934bf613-d52b-4c98-83df-bbe022881572-kube-api-access-tk8r8\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:20.818885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.818881    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934bf613-d52b-4c98-83df-bbe022881572-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:20.919731 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:20.919701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:21.019853 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.019814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:21.422385 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.422356    2580 generic.go:356] "Generic (PLEG): container finished" podID="934bf613-d52b-4c98-83df-bbe022881572" containerID="11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a" exitCode=0
Sep 18 18:26:21.422785 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.422400    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"
Sep 18 18:26:21.422785 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.422445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld" event={"ID":"934bf613-d52b-4c98-83df-bbe022881572","Type":"ContainerDied","Data":"11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a"}
Sep 18 18:26:21.422785 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.422492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld" event={"ID":"934bf613-d52b-4c98-83df-bbe022881572","Type":"ContainerDied","Data":"375f97e186b4bc71d8b560c4201895d264ed559c5375f9d711d32f0e71ba107b"}
Sep 18 18:26:21.422785 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.422514    2580 scope.go:117] "RemoveContainer" containerID="11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a"
Sep 18 18:26:21.432399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.432375    2580 scope.go:117] "RemoveContainer" containerID="11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a"
Sep 18 18:26:21.432692 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:26:21.432670    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a\": container with ID starting with 11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a not found: ID does not exist" containerID="11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a"
Sep 18 18:26:21.432778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.432701    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a"} err="failed to get container status \"11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a\": rpc error: code = NotFound desc = could not find container \"11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a\": container with ID starting with 11834b9c4fd10d9652dca462a5ba0e92f605a3ffcd2c4fd12973c593d9da977a not found: ID does not exist"
Sep 18 18:26:21.456670 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.456647    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"]
Sep 18 18:26:21.461623 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.461604    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f0cb-786765bc9d-r6dld"]
Sep 18 18:26:21.689782 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.689700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:21.910123 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:21.910091    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934bf613-d52b-4c98-83df-bbe022881572" path="/var/lib/kubelet/pods/934bf613-d52b-4c98-83df-bbe022881572/volumes"
Sep 18 18:26:22.379924 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:22.379888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:23.043164 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:23.043130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:23.680348 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:23.680313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:27.074236 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:27.074205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:27.189536 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:27.189504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:27.365478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:27.365395    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"]
Sep 18 18:26:27.365650 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:27.365626    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy" containerID="cri-o://9ff958d8ab1984e6cf3fc1c1903d83d6cfbe8a49687207aecb8d60632c758eff" gracePeriod=30
Sep 18 18:26:27.388895 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:27.388855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:27.881376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:27.881338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:28.636840 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:28.636795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:29.405196 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:29.405163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:30.207280 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:30.207243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:30.793763 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:30.793705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:31.473622 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:31.473575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:32.484867 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:32.484834    2580 generic.go:356] "Generic (PLEG): container finished" podID="f378c09a-0272-4a54-b476-02721bc3272a" containerID="9ff958d8ab1984e6cf3fc1c1903d83d6cfbe8a49687207aecb8d60632c758eff" exitCode=0
Sep 18 18:26:32.485309 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:32.484918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" event={"ID":"f378c09a-0272-4a54-b476-02721bc3272a","Type":"ContainerDied","Data":"9ff958d8ab1984e6cf3fc1c1903d83d6cfbe8a49687207aecb8d60632c758eff"}
Sep 18 18:26:32.485309 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:32.485050    2580 scope.go:117] "RemoveContainer" containerID="85781297e59a37aa06f58a9451f2ca4340bce2212d9d2731cba5a360f89587fc"
Sep 18 18:26:33.024563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.024539    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:26:33.126300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-istio-token\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126312    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-credential-socket\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-socket\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-data\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f378c09a-0272-4a54-b476-02721bc3272a-istiod-ca-cert\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f378c09a-0272-4a54-b476-02721bc3272a-istio-podinfo\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zds45\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-kube-api-access-zds45\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:33.126714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126627    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-certs\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:33.126990 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-envoy\") pod \"f378c09a-0272-4a54-b476-02721bc3272a\" (UID: \"f378c09a-0272-4a54-b476-02721bc3272a\") "
Sep 18 18:26:33.126990 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.126718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:33.127118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.127015    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.127118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.127050    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.127118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.127064    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.127118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.127060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f378c09a-0272-4a54-b476-02721bc3272a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:33.127501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.127477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:33.128879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.128854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:33.128879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.128863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f378c09a-0272-4a54-b476-02721bc3272a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:26:33.129149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.129121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-kube-api-access-zds45" (OuterVolumeSpecName: "kube-api-access-zds45") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "kube-api-access-zds45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:33.129469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.129452    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f378c09a-0272-4a54-b476-02721bc3272a" (UID: "f378c09a-0272-4a54-b476-02721bc3272a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:33.227753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.227727    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.227871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.227758    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.227871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.227772    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f378c09a-0272-4a54-b476-02721bc3272a-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.227871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.227784    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f378c09a-0272-4a54-b476-02721bc3272a-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.227871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.227798    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zds45\" (UniqueName: \"kubernetes.io/projected/f378c09a-0272-4a54-b476-02721bc3272a-kube-api-access-zds45\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.227871 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.227810    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f378c09a-0272-4a54-b476-02721bc3272a-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:33.491876 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.491794    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"
Sep 18 18:26:33.492335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.491799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh" event={"ID":"f378c09a-0272-4a54-b476-02721bc3272a","Type":"ContainerDied","Data":"ec66ade5a038675f18063065382846e4362c5eb046ff07622f193d768f29132e"}
Sep 18 18:26:33.492335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.491932    2580 scope.go:117] "RemoveContainer" containerID="9ff958d8ab1984e6cf3fc1c1903d83d6cfbe8a49687207aecb8d60632c758eff"
Sep 18 18:26:33.514012 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.513981    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"]
Sep 18 18:26:33.517651 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.517630    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-29f0cb-openshift-default-767f888db-7fkzh"]
Sep 18 18:26:33.911435 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:33.911407    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f378c09a-0272-4a54-b476-02721bc3272a" path="/var/lib/kubelet/pods/f378c09a-0272-4a54-b476-02721bc3272a/volumes"
Sep 18 18:26:38.219751 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.219688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:38.249906 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.249874    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"]
Sep 18 18:26:38.250160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.250133    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt" podUID="4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" containerName="payload-processing" containerID="cri-o://0101f50a7c71aab3c899d343474f200a6db1cd47fe561777291c02f48ba2dc43" gracePeriod=30
Sep 18 18:26:38.258599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.258107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"]
Sep 18 18:26:38.258599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.258325    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p" podUID="04cf9bfe-97b6-466d-89d3-29b7542a8300" containerName="payload-pre-processing" containerID="cri-o://cd69a4cfc1ef2401a250231634f2e5852dc026ab38a8ab5a4a887bf18d0c7cab" gracePeriod=30
Sep 18 18:26:38.294020 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.293981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:38.353505 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.353481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:38.441274 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.441240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:38.500411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.496847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:38.527778 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.527742    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" containerID="0101f50a7c71aab3c899d343474f200a6db1cd47fe561777291c02f48ba2dc43" exitCode=0
Sep 18 18:26:38.527958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.527849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt" event={"ID":"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e","Type":"ContainerDied","Data":"0101f50a7c71aab3c899d343474f200a6db1cd47fe561777291c02f48ba2dc43"}
Sep 18 18:26:38.527958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.527899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt" event={"ID":"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e","Type":"ContainerDied","Data":"f5bf1e72c730c9815df5aadd5b9cd1ecee8883250524578019432c005e69d69e"}
Sep 18 18:26:38.527958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.527922    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5bf1e72c730c9815df5aadd5b9cd1ecee8883250524578019432c005e69d69e"
Sep 18 18:26:38.528714 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.528423    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:26:38.530467 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.530443    2580 generic.go:356] "Generic (PLEG): container finished" podID="04cf9bfe-97b6-466d-89d3-29b7542a8300" containerID="cd69a4cfc1ef2401a250231634f2e5852dc026ab38a8ab5a4a887bf18d0c7cab" exitCode=0
Sep 18 18:26:38.530586 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.530571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p" event={"ID":"04cf9bfe-97b6-466d-89d3-29b7542a8300","Type":"ContainerDied","Data":"cd69a4cfc1ef2401a250231634f2e5852dc026ab38a8ab5a4a887bf18d0c7cab"}
Sep 18 18:26:38.541294 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.541273    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:26:38.680589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.680557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf9bfe-97b6-466d-89d3-29b7542a8300-plugins-config-volume\") pod \"04cf9bfe-97b6-466d-89d3-29b7542a8300\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") "
Sep 18 18:26:38.680788 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.680627    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtt6n\" (UniqueName: \"kubernetes.io/projected/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-kube-api-access-qtt6n\") pod \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") "
Sep 18 18:26:38.680788 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.680675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-plugins-config-volume\") pod \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\" (UID: \"4f0d503f-5a10-465c-b0b2-abe2bdc1c54e\") "
Sep 18 18:26:38.680788 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.680719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v5xg\" (UniqueName: \"kubernetes.io/projected/04cf9bfe-97b6-466d-89d3-29b7542a8300-kube-api-access-6v5xg\") pod \"04cf9bfe-97b6-466d-89d3-29b7542a8300\" (UID: \"04cf9bfe-97b6-466d-89d3-29b7542a8300\") "
Sep 18 18:26:38.681019 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.680991    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cf9bfe-97b6-466d-89d3-29b7542a8300-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04cf9bfe-97b6-466d-89d3-29b7542a8300" (UID: "04cf9bfe-97b6-466d-89d3-29b7542a8300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:38.681111 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.681014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" (UID: "4f0d503f-5a10-465c-b0b2-abe2bdc1c54e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:38.682937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.682916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-kube-api-access-qtt6n" (OuterVolumeSpecName: "kube-api-access-qtt6n") pod "4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" (UID: "4f0d503f-5a10-465c-b0b2-abe2bdc1c54e"). InnerVolumeSpecName "kube-api-access-qtt6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:38.682937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.682923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cf9bfe-97b6-466d-89d3-29b7542a8300-kube-api-access-6v5xg" (OuterVolumeSpecName: "kube-api-access-6v5xg") pod "04cf9bfe-97b6-466d-89d3-29b7542a8300" (UID: "04cf9bfe-97b6-466d-89d3-29b7542a8300"). InnerVolumeSpecName "kube-api-access-6v5xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:38.781945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.781862    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:38.781945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.781891    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v5xg\" (UniqueName: \"kubernetes.io/projected/04cf9bfe-97b6-466d-89d3-29b7542a8300-kube-api-access-6v5xg\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:38.781945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.781900    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf9bfe-97b6-466d-89d3-29b7542a8300-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:38.781945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.781908    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtt6n\" (UniqueName: \"kubernetes.io/projected/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e-kube-api-access-qtt6n\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:38.822849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.822812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:38.922817 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:38.922785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:39.537022 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.536935    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"
Sep 18 18:26:39.537022 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.536984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p" event={"ID":"04cf9bfe-97b6-466d-89d3-29b7542a8300","Type":"ContainerDied","Data":"0d50183edec90892204ba2aa401d36eb6ac37e61bddb30b6cc51051f11c0ffb3"}
Sep 18 18:26:39.537507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.537065    2580 scope.go:117] "RemoveContainer" containerID="cd69a4cfc1ef2401a250231634f2e5852dc026ab38a8ab5a4a887bf18d0c7cab"
Sep 18 18:26:39.537507 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.537197    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"
Sep 18 18:26:39.560182 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.560157    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"]
Sep 18 18:26:39.564369 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.564349    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5e6d4-78c6696df8-2fw5p"]
Sep 18 18:26:39.574102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.574083    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"]
Sep 18 18:26:39.577905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.577885    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5e6d4-7bbfb4c55-778vt"]
Sep 18 18:26:39.710180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.710151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:39.911376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.911347    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04cf9bfe-97b6-466d-89d3-29b7542a8300" path="/var/lib/kubelet/pods/04cf9bfe-97b6-466d-89d3-29b7542a8300/volumes"
Sep 18 18:26:39.911771 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:39.911755    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" path="/var/lib/kubelet/pods/4f0d503f-5a10-465c-b0b2-abe2bdc1c54e/volumes"
Sep 18 18:26:40.374917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:40.374865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:41.306506 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:41.306459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:42.533488 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:42.533432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:43.614774 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:43.614741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:44.550621 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:44.550568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:44.651666 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:44.651628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:44.815751 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:44.815672    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"]
Sep 18 18:26:44.815930 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:44.815905    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy" containerID="cri-o://4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad" gracePeriod=30
Sep 18 18:26:44.841110 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:44.841079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:45.530933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:45.530893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:46.260545 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:46.260507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:47.106656 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:47.106609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:47.747270 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:47.747235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:48.389995 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:48.389964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:50.069471 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.069448    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:26:50.085605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d3295f-369d-4e08-b165-3b94581287eb-istio-podinfo\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqbgp\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-kube-api-access-dqbgp\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085741 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-istio-token\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085817 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-envoy\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085857 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-data\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085904 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-credential-socket\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-socket\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.085974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d3295f-369d-4e08-b165-3b94581287eb-istiod-ca-cert\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.086104 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.085986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-certs\") pod \"f5d3295f-369d-4e08-b165-3b94581287eb\" (UID: \"f5d3295f-369d-4e08-b165-3b94581287eb\") "
Sep 18 18:26:50.086162 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.086124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:50.086443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.086233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:50.086608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.086414    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.086608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.086247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:50.086755 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.086517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:50.086755 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.086559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d3295f-369d-4e08-b165-3b94581287eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:50.087880 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.087858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5d3295f-369d-4e08-b165-3b94581287eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:26:50.088075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.088048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:26:50.088537 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.088497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-kube-api-access-dqbgp" (OuterVolumeSpecName: "kube-api-access-dqbgp") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "kube-api-access-dqbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:50.088725 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.088702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5d3295f-369d-4e08-b165-3b94581287eb" (UID: "f5d3295f-369d-4e08-b165-3b94581287eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:50.187399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187356    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d3295f-369d-4e08-b165-3b94581287eb-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187391    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqbgp\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-kube-api-access-dqbgp\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187405    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d3295f-369d-4e08-b165-3b94581287eb-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187399 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187414    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187421    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187429    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187438    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d3295f-369d-4e08-b165-3b94581287eb-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.187695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.187446    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d3295f-369d-4e08-b165-3b94581287eb-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:26:50.377101 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.376994    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"]
Sep 18 18:26:50.377468 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy"
Sep 18 18:26:50.377515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377472    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy"
Sep 18 18:26:50.377515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377482    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90e817a2-893c-4802-8304-ec3e5ab690ca" containerName="maas-api"
Sep 18 18:26:50.377515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377488    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="90e817a2-893c-4802-8304-ec3e5ab690ca" containerName="maas-api"
Sep 18 18:26:50.377515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377497    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cf9bfe-97b6-466d-89d3-29b7542a8300" containerName="payload-pre-processing"
Sep 18 18:26:50.377515 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377503    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cf9bfe-97b6-466d-89d3-29b7542a8300" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377522    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" containerName="payload-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377527    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" containerName="payload-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377535    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377540    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934bf613-d52b-4c98-83df-bbe022881572" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377550    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="934bf613-d52b-4c98-83df-bbe022881572" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377564    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27953c2f-224c-437b-bb29-62c6126df108" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377576    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27953c2f-224c-437b-bb29-62c6126df108" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377635    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377646    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cf9bfe-97b6-466d-89d3-29b7542a8300" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377656    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377666    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377676    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f378c09a-0272-4a54-b476-02721bc3272a" containerName="istio-proxy"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27953c2f-224c-437b-bb29-62c6126df108" containerName="payload-pre-processing"
Sep 18 18:26:50.377687 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377695    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0d503f-5a10-465c-b0b2-abe2bdc1c54e" containerName="payload-processing"
Sep 18 18:26:50.378459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377705    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="90e817a2-893c-4802-8304-ec3e5ab690ca" containerName="maas-api"
Sep 18 18:26:50.378459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.377712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="934bf613-d52b-4c98-83df-bbe022881572" containerName="payload-pre-processing"
Sep 18 18:26:50.381299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.381269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.383483 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.383450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-011541\"/\"default-dockercfg-zdxr6\""
Sep 18 18:26:50.383628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.383507    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-011541\"/\"rate-test-model-011541-kserve-self-signed-certs\""
Sep 18 18:26:50.384057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.384015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-011541\"/\"kube-root-ca.crt\""
Sep 18 18:26:50.384142 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.384076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-011541\"/\"openshift-service-ca.crt\""
Sep 18 18:26:50.389521 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.389491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"]
Sep 18 18:26:50.404158 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.404121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:50.490355 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.490317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-model-cache\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.490538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.490371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tmp-dir\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.490538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.490428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-dshm\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.490538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.490463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs662\" (UniqueName: \"kubernetes.io/projected/bd97dec6-3764-4f2c-badd-3d9062bb75bf-kube-api-access-hs662\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.490538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.490489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-home\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.490538 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.490533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tls-certs\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.591701 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.591655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tls-certs\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.591915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.591714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-model-cache\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.591915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.591764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tmp-dir\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.591915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.591802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-dshm\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.591915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.591852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs662\" (UniqueName: \"kubernetes.io/projected/bd97dec6-3764-4f2c-badd-3d9062bb75bf-kube-api-access-hs662\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.591915 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.591885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-home\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.592246 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.592220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-model-cache\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.592421 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.592400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tmp-dir\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.592526 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.592503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-home\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.594485 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.594462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-dshm\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.594595 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.594545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tls-certs\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.598599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.598572    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5d3295f-369d-4e08-b165-3b94581287eb" containerID="4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad" exitCode=0
Sep 18 18:26:50.598718 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.598619    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"
Sep 18 18:26:50.598718 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.598628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" event={"ID":"f5d3295f-369d-4e08-b165-3b94581287eb","Type":"ContainerDied","Data":"4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad"}
Sep 18 18:26:50.598718 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.598686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv" event={"ID":"f5d3295f-369d-4e08-b165-3b94581287eb","Type":"ContainerDied","Data":"924a729dcbb7739b95b5a8148139439349959e330f17cd395a1095a9ec5a7d55"}
Sep 18 18:26:50.598718 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.598716    2580 scope.go:117] "RemoveContainer" containerID="4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad"
Sep 18 18:26:50.599416 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.599394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs662\" (UniqueName: \"kubernetes.io/projected/bd97dec6-3764-4f2c-badd-3d9062bb75bf-kube-api-access-hs662\") pod \"rate-test-model-011541-kserve-578fd8d899-nswbt\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") " pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.607910 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.607887    2580 scope.go:117] "RemoveContainer" containerID="733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"
Sep 18 18:26:50.616873 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.616831    2580 scope.go:117] "RemoveContainer" containerID="4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad"
Sep 18 18:26:50.617345 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:26:50.617315    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad\": container with ID starting with 4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad not found: ID does not exist" containerID="4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad"
Sep 18 18:26:50.617474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.617357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad"} err="failed to get container status \"4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad\": rpc error: code = NotFound desc = could not find container \"4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad\": container with ID starting with 4b248342f4f12637a9aebd6db6189626570ff59f79ac5bf7a42428e2cef8c9ad not found: ID does not exist"
Sep 18 18:26:50.617474 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.617381    2580 scope.go:117] "RemoveContainer" containerID="733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"
Sep 18 18:26:50.617770 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:26:50.617752    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0\": container with ID starting with 733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0 not found: ID does not exist" containerID="733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"
Sep 18 18:26:50.617814 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.617779    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0"} err="failed to get container status \"733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0\": rpc error: code = NotFound desc = could not find container \"733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0\": container with ID starting with 733f5e0e9de4a027043cd694b2928fc306a03dc4aa85ead6386c832db2eefaa0 not found: ID does not exist"
Sep 18 18:26:50.618841 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.618820    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"]
Sep 18 18:26:50.622457 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.622435    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5e6d4-openshift-default-5f787bb848-c8mkv"]
Sep 18 18:26:50.693885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.693843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:26:50.971180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:50.971158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"]
Sep 18 18:26:50.973720 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:26:50.973696    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd97dec6_3764_4f2c_badd_3d9062bb75bf.slice/crio-c967691a004da48c1a3ca7d7d197e11fa33d54751d40edd7983b262c999c4984 WatchSource:0}: Error finding container c967691a004da48c1a3ca7d7d197e11fa33d54751d40edd7983b262c999c4984: Status 404 returned error can't find the container with id c967691a004da48c1a3ca7d7d197e11fa33d54751d40edd7983b262c999c4984
Sep 18 18:26:51.606346 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:51.606311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt" event={"ID":"bd97dec6-3764-4f2c-badd-3d9062bb75bf","Type":"ContainerStarted","Data":"0abe57d30ac18b448704875e32a8fbc136375616ac6ab5e4c31f6a0a92eff3cb"}
Sep 18 18:26:51.606346 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:51.606356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt" event={"ID":"bd97dec6-3764-4f2c-badd-3d9062bb75bf","Type":"ContainerStarted","Data":"c967691a004da48c1a3ca7d7d197e11fa33d54751d40edd7983b262c999c4984"}
Sep 18 18:26:51.622305 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:51.622258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt" podStartSLOduration=1.6222431959999999 podStartE2EDuration="1.622243196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:26:51.621519552 +0000 UTC m=+1581.773394386" watchObservedRunningTime="2026-09-18 18:26:51.622243196 +0000 UTC m=+1581.774118030"
Sep 18 18:26:51.911381 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:51.911351    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" path="/var/lib/kubelet/pods/f5d3295f-369d-4e08-b165-3b94581287eb/volumes"
Sep 18 18:26:53.255534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:53.255460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:53.312482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:53.312451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:55.744453 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:55.744418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:55.824456 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:55.821944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:55.911400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:55.911350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:55.974415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:55.974377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:56.349377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:56.349348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:56.450505 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:56.450467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:57.056798 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:57.056763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:57.703927 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:26:57.703896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:00.695006 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:00.694970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:27:00.695532 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:00.695060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:27:00.707582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:00.707558    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:27:01.669304 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:01.669274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:27:02.454916 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:02.454872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:02.510952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:02.510900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:02.697271 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:02.697236    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"]
Sep 18 18:27:02.697491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:02.697469    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" containerID="cri-o://03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b" gracePeriod=30
Sep 18 18:27:02.718244 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:02.718157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:07.800489 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:07.800454    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 18:27:07.800837 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:07.800521    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 18:27:07.958740 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:07.958680    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:27:08.055244 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-socket\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-certs\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74829c53-0621-427c-972e-6572dd997284-istio-podinfo\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-data\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-istio-token\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74829c53-0621-427c-972e-6572dd997284-istiod-ca-cert\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055425 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055395    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-envoy\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-credential-socket\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h9lw\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-kube-api-access-6h9lw\") pod \"74829c53-0621-427c-972e-6572dd997284\" (UID: \"74829c53-0621-427c-972e-6572dd997284\") "
Sep 18 18:27:08.055732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:27:08.055732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-data" (OuterVolumeSpecName: "istio-data") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:27:08.055952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:27:08.055952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055807    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:27:08.055952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74829c53-0621-427c-972e-6572dd997284-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:27:08.055952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055930    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.055952 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055949    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.056229 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055965    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.056229 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.055977    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.057656 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.057634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-istio-token" (OuterVolumeSpecName: "istio-token") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:27:08.057864 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.057844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-kube-api-access-6h9lw" (OuterVolumeSpecName: "kube-api-access-6h9lw") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "kube-api-access-6h9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:27:08.058194 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.058169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74829c53-0621-427c-972e-6572dd997284-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:27:08.058289 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.058186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74829c53-0621-427c-972e-6572dd997284" (UID: "74829c53-0621-427c-972e-6572dd997284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:27:08.156482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.156442    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74829c53-0621-427c-972e-6572dd997284-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.156482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.156473    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.156482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.156482    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74829c53-0621-427c-972e-6572dd997284-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.156482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.156490    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74829c53-0621-427c-972e-6572dd997284-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.156758 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.156498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9lw\" (UniqueName: \"kubernetes.io/projected/74829c53-0621-427c-972e-6572dd997284-kube-api-access-6h9lw\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:27:08.692743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.692700    2580 generic.go:356] "Generic (PLEG): container finished" podID="74829c53-0621-427c-972e-6572dd997284" containerID="03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b" exitCode=0
Sep 18 18:27:08.692920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.692753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"
Sep 18 18:27:08.692920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.692789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" event={"ID":"74829c53-0621-427c-972e-6572dd997284","Type":"ContainerDied","Data":"03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b"}
Sep 18 18:27:08.692920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.692844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc" event={"ID":"74829c53-0621-427c-972e-6572dd997284","Type":"ContainerDied","Data":"a4bf9c6f674800e4f6e6a8b775a69de1e45bcc09a0a4bda123c950ca9f1cf514"}
Sep 18 18:27:08.692920 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.692869    2580 scope.go:117] "RemoveContainer" containerID="03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b"
Sep 18 18:27:08.702394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.702374    2580 scope.go:117] "RemoveContainer" containerID="8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"
Sep 18 18:27:08.710514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.710494    2580 scope.go:117] "RemoveContainer" containerID="03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b"
Sep 18 18:27:08.710777 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:27:08.710757    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b\": container with ID starting with 03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b not found: ID does not exist" containerID="03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b"
Sep 18 18:27:08.710834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.710785    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b"} err="failed to get container status \"03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b\": rpc error: code = NotFound desc = could not find container \"03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b\": container with ID starting with 03969824d051ef4daf1eec2b80308c9d1e0a3fb3d95165704cd11b6344e45a1b not found: ID does not exist"
Sep 18 18:27:08.710834 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.710801    2580 scope.go:117] "RemoveContainer" containerID="8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"
Sep 18 18:27:08.711059 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:27:08.711041    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3\": container with ID starting with 8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3 not found: ID does not exist" containerID="8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"
Sep 18 18:27:08.711106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.711065    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3"} err="failed to get container status \"8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3\": rpc error: code = NotFound desc = could not find container \"8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3\": container with ID starting with 8f8e24a8ededc2b7123d13b20974febafe0608b0ce87b8709ac3add4f66b38f3 not found: ID does not exist"
Sep 18 18:27:08.713752 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.713718    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"]
Sep 18 18:27:08.719801 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:08.719779    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1f7000-openshift-default-5846598874-2bmvc"]
Sep 18 18:27:09.910509 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:09.910476    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74829c53-0621-427c-972e-6572dd997284" path="/var/lib/kubelet/pods/74829c53-0621-427c-972e-6572dd997284/volumes"
Sep 18 18:27:17.813487 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:17.813401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:32.141262 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:32.141230    2580 scope.go:117] "RemoveContainer" containerID="0101f50a7c71aab3c899d343474f200a6db1cd47fe561777291c02f48ba2dc43"
Sep 18 18:27:52.843576 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:52.843544    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"]
Sep 18 18:27:52.862688 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:52.862656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:53.809329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:53.809286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:54.270692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.270660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:54.533486 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.533414    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"]
Sep 18 18:27:54.533978 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.533963    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy"
Sep 18 18:27:54.534081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.533981    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy"
Sep 18 18:27:54.534081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.534005    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy"
Sep 18 18:27:54.534081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.534015    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d3295f-369d-4e08-b165-3b94581287eb" containerName="istio-proxy"
Sep 18 18:27:54.534081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.534043    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy"
Sep 18 18:27:54.534081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.534052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy"
Sep 18 18:27:54.534239 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.534154    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy"
Sep 18 18:27:54.534239 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.534168    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74829c53-0621-427c-972e-6572dd997284" containerName="istio-proxy"
Sep 18 18:27:54.537527 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.537503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.539602 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.539583    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-15e8cc-openshift-default-dockercfg-6grp2\""
Sep 18 18:27:54.547756 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.547730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"]
Sep 18 18:27:54.558847 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.558825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:54.675062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-podinfo\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee0eb77-299e-4050-b23b-7ef359bc26bb-istiod-ca-cert\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-socket\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-envoy\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-credential-socket\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-data\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-certs\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfksw\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-kube-api-access-qfksw\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.675394 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.675375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-token\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775768 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfksw\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-kube-api-access-qfksw\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-token\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-podinfo\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee0eb77-299e-4050-b23b-7ef359bc26bb-istiod-ca-cert\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-socket\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-envoy\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.775917 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-credential-socket\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-data\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.775986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-certs\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776470 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.776371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-socket\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776470 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.776387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-certs\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776591 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.776556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-credential-socket\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.776633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee0eb77-299e-4050-b23b-7ef359bc26bb-istiod-ca-cert\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.776711 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.776660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-data\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.778564 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.778540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-envoy\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.778811 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.778794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-podinfo\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.783568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.783514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfksw\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-kube-api-access-qfksw\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.783804 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.783787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-token\") pod \"e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") " pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:54.848872 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:54.848842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:55.064100 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.063986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:55.122858 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.122836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"]
Sep 18 18:27:55.124798 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:27:55.124768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee0eb77_299e_4050_b23b_7ef359bc26bb.slice/crio-d0ba093d77975a7332440449533aa733439847f92753b8b6a4e1ad0775b3b725 WatchSource:0}: Error finding container d0ba093d77975a7332440449533aa733439847f92753b8b6a4e1ad0775b3b725: Status 404 returned error can't find the container with id d0ba093d77975a7332440449533aa733439847f92753b8b6a4e1ad0775b3b725
Sep 18 18:27:55.127409 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.127367    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:27:55.127530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.127459    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:27:55.127530 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.127507    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:27:55.933958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.933919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" event={"ID":"eee0eb77-299e-4050-b23b-7ef359bc26bb","Type":"ContainerStarted","Data":"2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"}
Sep 18 18:27:55.934347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.933976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" event={"ID":"eee0eb77-299e-4050-b23b-7ef359bc26bb","Type":"ContainerStarted","Data":"d0ba093d77975a7332440449533aa733439847f92753b8b6a4e1ad0775b3b725"}
Sep 18 18:27:55.953890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:55.953841    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podStartSLOduration=1.953824693 podStartE2EDuration="1.953824693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:27:55.952732962 +0000 UTC m=+1646.104607797" watchObservedRunningTime="2026-09-18 18:27:55.953824693 +0000 UTC m=+1646.105699528"
Sep 18 18:27:56.849435 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:56.849394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:56.854781 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:56.854755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:56.939510 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:56.939474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:56.940379 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:56.940358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:27:57.115383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:57.115297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:57.808300 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:57.808260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:57.906519 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:57.906466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:58.007837 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:58.007801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:58.710484 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:58.710448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:59.470251 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.470220    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"]
Sep 18 18:27:59.479123 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.479089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.480946 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.480927    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-15e8cc-serving-cert\""
Sep 18 18:27:59.485656 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.485630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"]
Sep 18 18:27:59.512053 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.512012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:59.612649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.612620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:59.622331 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.622313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.622440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.622341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.622440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.622382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53965664-7b53-42e6-a85d-da75cc58acd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.622523 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.622441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpmb\" (UniqueName: \"kubernetes.io/projected/53965664-7b53-42e6-a85d-da75cc58acd8-kube-api-access-4gpmb\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.713069 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.713007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:59.723220 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.723152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.723220 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.723196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.723395 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.723265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53965664-7b53-42e6-a85d-da75cc58acd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.723395 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.723327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpmb\" (UniqueName: \"kubernetes.io/projected/53965664-7b53-42e6-a85d-da75cc58acd8-kube-api-access-4gpmb\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.723912 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.723893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53965664-7b53-42e6-a85d-da75cc58acd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.725770 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.725752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.725867 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.725793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.731534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.731510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpmb\" (UniqueName: \"kubernetes.io/projected/53965664-7b53-42e6-a85d-da75cc58acd8-kube-api-access-4gpmb\") pod \"maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.792852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.792823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:27:59.970850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:27:59.970814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:00.064625 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:00.064542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:00.066062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:00.066016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"]
Sep 18 18:28:00.068201 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:28:00.068178    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53965664_7b53_42e6_a85d_da75cc58acd8.slice/crio-298ddb33dbef805b025f4a5a5ae9169f5536097d63b23c5553af1f34479a90f2 WatchSource:0}: Error finding container 298ddb33dbef805b025f4a5a5ae9169f5536097d63b23c5553af1f34479a90f2: Status 404 returned error can't find the container with id 298ddb33dbef805b025f4a5a5ae9169f5536097d63b23c5553af1f34479a90f2
Sep 18 18:28:00.961101 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:00.961069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5" event={"ID":"53965664-7b53-42e6-a85d-da75cc58acd8","Type":"ContainerStarted","Data":"cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202"}
Sep 18 18:28:00.961481 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:00.961118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5" event={"ID":"53965664-7b53-42e6-a85d-da75cc58acd8","Type":"ContainerStarted","Data":"298ddb33dbef805b025f4a5a5ae9169f5536097d63b23c5553af1f34479a90f2"}
Sep 18 18:28:00.961481 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:00.961148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:28:00.976958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:00.976913    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5" podStartSLOduration=1.539824484 podStartE2EDuration="1.976899205s" totalImagesPullingTime="437.074721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:28:00.976194645 +0000 UTC m=+1651.128069481" watchObservedRunningTime="2026-09-18 18:28:00.976899205 +0000 UTC m=+1651.128774039"
Sep 18 18:28:06.972131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:06.972099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:28:07.862972 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:07.862931    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt" podUID="bd97dec6-3764-4f2c-badd-3d9062bb75bf" containerName="main" containerID="cri-o://0abe57d30ac18b448704875e32a8fbc136375616ac6ab5e4c31f6a0a92eff3cb" gracePeriod=45
Sep 18 18:28:07.995660 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:07.995629    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd97dec6-3764-4f2c-badd-3d9062bb75bf" containerID="0abe57d30ac18b448704875e32a8fbc136375616ac6ab5e4c31f6a0a92eff3cb" exitCode=0
Sep 18 18:28:07.995998 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:07.995714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt" event={"ID":"bd97dec6-3764-4f2c-badd-3d9062bb75bf","Type":"ContainerDied","Data":"0abe57d30ac18b448704875e32a8fbc136375616ac6ab5e4c31f6a0a92eff3cb"}
Sep 18 18:28:08.146933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.146607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:28:08.206157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs662\" (UniqueName: \"kubernetes.io/projected/bd97dec6-3764-4f2c-badd-3d9062bb75bf-kube-api-access-hs662\") pod \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") "
Sep 18 18:28:08.206157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-home\") pod \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") "
Sep 18 18:28:08.206405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-model-cache\") pod \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") "
Sep 18 18:28:08.206405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206200    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-dshm\") pod \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") "
Sep 18 18:28:08.206405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206266    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tls-certs\") pod \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") "
Sep 18 18:28:08.206405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tmp-dir\") pod \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\" (UID: \"bd97dec6-3764-4f2c-badd-3d9062bb75bf\") "
Sep 18 18:28:08.206618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-home" (OuterVolumeSpecName: "home") pod "bd97dec6-3764-4f2c-badd-3d9062bb75bf" (UID: "bd97dec6-3764-4f2c-badd-3d9062bb75bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:28:08.206618 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd97dec6-3764-4f2c-badd-3d9062bb75bf" (UID: "bd97dec6-3764-4f2c-badd-3d9062bb75bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:28:08.206773 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd97dec6-3764-4f2c-badd-3d9062bb75bf" (UID: "bd97dec6-3764-4f2c-badd-3d9062bb75bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:28:08.206773 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206711    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:28:08.206773 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.206731    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:28:08.208852 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.208823    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd97dec6-3764-4f2c-badd-3d9062bb75bf-kube-api-access-hs662" (OuterVolumeSpecName: "kube-api-access-hs662") pod "bd97dec6-3764-4f2c-badd-3d9062bb75bf" (UID: "bd97dec6-3764-4f2c-badd-3d9062bb75bf"). InnerVolumeSpecName "kube-api-access-hs662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:28:08.209075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.209017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd97dec6-3764-4f2c-badd-3d9062bb75bf" (UID: "bd97dec6-3764-4f2c-badd-3d9062bb75bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:28:08.209075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.209015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-dshm" (OuterVolumeSpecName: "dshm") pod "bd97dec6-3764-4f2c-badd-3d9062bb75bf" (UID: "bd97dec6-3764-4f2c-badd-3d9062bb75bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:28:08.307486 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.307456    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs662\" (UniqueName: \"kubernetes.io/projected/bd97dec6-3764-4f2c-badd-3d9062bb75bf-kube-api-access-hs662\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:28:08.307486 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.307482    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:28:08.307486 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.307492    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:28:08.307704 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:08.307501    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd97dec6-3764-4f2c-badd-3d9062bb75bf-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:28:09.001979 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:09.001946    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"
Sep 18 18:28:09.001979 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:09.001963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt" event={"ID":"bd97dec6-3764-4f2c-badd-3d9062bb75bf","Type":"ContainerDied","Data":"c967691a004da48c1a3ca7d7d197e11fa33d54751d40edd7983b262c999c4984"}
Sep 18 18:28:09.002555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:09.002044    2580 scope.go:117] "RemoveContainer" containerID="0abe57d30ac18b448704875e32a8fbc136375616ac6ab5e4c31f6a0a92eff3cb"
Sep 18 18:28:09.022090 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:09.022060    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"]
Sep 18 18:28:09.026886 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:09.026865    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-011541/rate-test-model-011541-kserve-578fd8d899-nswbt"]
Sep 18 18:28:09.911678 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:09.911643    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd97dec6-3764-4f2c-badd-3d9062bb75bf" path="/var/lib/kubelet/pods/bd97dec6-3764-4f2c-badd-3d9062bb75bf/volumes"
Sep 18 18:28:12.058636 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:12.057601    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:50312->10.134.0.64:15021: read: connection reset by peer" start-of-body=
Sep 18 18:28:12.058636 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:12.057661    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:50312->10.134.0.64:15021: read: connection reset by peer"
Sep 18 18:28:13.022482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:13.022451    2580 generic.go:356] "Generic (PLEG): container finished" podID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerID="2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7" exitCode=0
Sep 18 18:28:13.022654 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:13.022511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" event={"ID":"eee0eb77-299e-4050-b23b-7ef359bc26bb","Type":"ContainerDied","Data":"2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"}
Sep 18 18:28:13.022855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:13.022838    2580 scope.go:117] "RemoveContainer" containerID="2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"
Sep 18 18:28:13.023955 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:13.023918    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:13.024015 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:13.023990    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:13.024070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:13.024021    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:14.033056 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:14.033000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" event={"ID":"eee0eb77-299e-4050-b23b-7ef359bc26bb","Type":"ContainerStarted","Data":"a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892"}
Sep 18 18:28:14.849941 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:14.849896    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:28:14.850329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:14.850303    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 18:28:14.850558 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:14.850513    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 18:28:15.850216 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:15.850169    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 18:28:15.850609 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:15.850237    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 18:28:16.850078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:16.850008    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 18:28:16.850262 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:16.850146    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 18:28:17.850076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:17.850006    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 18:28:17.850270 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:17.850106    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 18:28:18.850270 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:18.850235    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 18:28:18.850697 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:18.850295    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 18:28:19.852589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:19.852551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:28:19.853062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:19.852861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:28:19.853503 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:19.853484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:28:21.444331 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:21.444292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:22.217791 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:22.217753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:22.755380 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:22.755342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:22.855700 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:22.855664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:23.487439 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.487398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:23.587633 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.587587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:23.789102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.788999    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"]
Sep 18 18:28:23.789636 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.789613    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97dec6-3764-4f2c-badd-3d9062bb75bf" containerName="main"
Sep 18 18:28:23.789690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.789643    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97dec6-3764-4f2c-badd-3d9062bb75bf" containerName="main"
Sep 18 18:28:23.789794 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.789782    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd97dec6-3764-4f2c-badd-3d9062bb75bf" containerName="main"
Sep 18 18:28:23.793210 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.793176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.795230 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.795213    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-46c63c-openshift-default-dockercfg-qgjl7\""
Sep 18 18:28:23.806168 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.806144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"]
Sep 18 18:28:23.847797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.847769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-podinfo\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.847907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.847805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-data\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.847907 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.847885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-socket\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.847999 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.847950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-credential-socket\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.848116 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.848094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8b84\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-kube-api-access-d8b84\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.848195 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.848165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istiod-ca-cert\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.848261 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.848199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-certs\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.848529 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.848224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-envoy\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.848638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.848566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-token\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.871166 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.871133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:23.949921 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.949894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istiod-ca-cert\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.949924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-certs\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.949946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-envoy\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.949976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-token\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-podinfo\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-data\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-socket\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-credential-socket\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950361 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8b84\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-kube-api-access-d8b84\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950518 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-certs\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950660 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-data\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-socket\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istiod-ca-cert\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.950845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.950804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-credential-socket\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.952885 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.952853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-envoy\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.953014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.952994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-podinfo\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.961565 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.961544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-token\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.962403 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.962379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8b84\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-kube-api-access-d8b84\") pod \"e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") " pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:23.986420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:23.986393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:24.105133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:24.105068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:24.278125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:24.278078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:24.378748 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:24.378721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"]
Sep 18 18:28:24.380580 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:28:24.380544    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab1e956_1d53_46ec_8b80_d40a22cff42d.slice/crio-152fdd04e638143886c67f0918b506d1c6efe2e9f0c911c2e28890379f6f336c WatchSource:0}: Error finding container 152fdd04e638143886c67f0918b506d1c6efe2e9f0c911c2e28890379f6f336c: Status 404 returned error can't find the container with id 152fdd04e638143886c67f0918b506d1c6efe2e9f0c911c2e28890379f6f336c
Sep 18 18:28:24.382948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:24.382918    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:24.383047 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:24.382982    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:24.383047 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:24.383009    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:25.092260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:25.092229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" event={"ID":"4ab1e956-1d53-46ec-8b80-d40a22cff42d","Type":"ContainerStarted","Data":"8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"}
Sep 18 18:28:25.092641 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:25.092276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" event={"ID":"4ab1e956-1d53-46ec-8b80-d40a22cff42d","Type":"ContainerStarted","Data":"152fdd04e638143886c67f0918b506d1c6efe2e9f0c911c2e28890379f6f336c"}
Sep 18 18:28:25.105638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:25.105616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:25.110289 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:25.110260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:25.112412 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:25.112369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podStartSLOduration=2.112354494 podStartE2EDuration="2.112354494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:28:25.111075745 +0000 UTC m=+1675.262950580" watchObservedRunningTime="2026-09-18 18:28:25.112354494 +0000 UTC m=+1675.264229328"
Sep 18 18:28:26.097157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:26.097128    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:26.098129 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:26.098107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:26.894495 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:26.894459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:27.190537 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:27.190503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:27.279177 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:27.279124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:27.379561 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:27.379526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:28.116081 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:28.116043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:29.266282 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:29.266239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:30.116299 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.116258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:30.219423 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.219384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:30.380565 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.380473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:30.583118 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.583084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:30.682756 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.682727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:30.730908 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.730877    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"]
Sep 18 18:28:30.734608 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.734591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:30.736958 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.736935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-46c63c\""
Sep 18 18:28:30.737085 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.736943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-46c63c-dockercfg-wk67l\""
Sep 18 18:28:30.744922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.744896    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"]
Sep 18 18:28:30.815090 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.815057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjz6\" (UniqueName: \"kubernetes.io/projected/52d48752-1d52-4836-a475-a7238c9791b6-kube-api-access-pqjz6\") pod \"payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:30.815275 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.815110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d48752-1d52-4836-a475-a7238c9791b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:30.915909 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.915873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjz6\" (UniqueName: \"kubernetes.io/projected/52d48752-1d52-4836-a475-a7238c9791b6-kube-api-access-pqjz6\") pod \"payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:30.916101 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.915920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d48752-1d52-4836-a475-a7238c9791b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:30.916575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.916556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d48752-1d52-4836-a475-a7238c9791b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:30.925619 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:30.925593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjz6\" (UniqueName: \"kubernetes.io/projected/52d48752-1d52-4836-a475-a7238c9791b6-kube-api-access-pqjz6\") pod \"payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:31.046232 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:31.046165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:31.334011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:31.333981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"]
Sep 18 18:28:31.336341 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:28:31.336308    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d48752_1d52_4836_a475_a7238c9791b6.slice/crio-eeb855df5f9ad73894e33a32f899dce3a2c9bce4900e04427ab15c812ac6a24d WatchSource:0}: Error finding container eeb855df5f9ad73894e33a32f899dce3a2c9bce4900e04427ab15c812ac6a24d: Status 404 returned error can't find the container with id eeb855df5f9ad73894e33a32f899dce3a2c9bce4900e04427ab15c812ac6a24d
Sep 18 18:28:32.127983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:32.127947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4" event={"ID":"52d48752-1d52-4836-a475-a7238c9791b6","Type":"ContainerStarted","Data":"b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6"}
Sep 18 18:28:32.128358 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:32.128048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4" event={"ID":"52d48752-1d52-4836-a475-a7238c9791b6","Type":"ContainerStarted","Data":"eeb855df5f9ad73894e33a32f899dce3a2c9bce4900e04427ab15c812ac6a24d"}
Sep 18 18:28:32.128358 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:32.128078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:32.148806 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:32.148749    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4" podStartSLOduration=2.148735609 podStartE2EDuration="2.148735609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:28:32.146781025 +0000 UTC m=+1682.298655863" watchObservedRunningTime="2026-09-18 18:28:32.148735609 +0000 UTC m=+1682.300610444"
Sep 18 18:28:39.390610 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:39.390575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:43.135426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:43.135338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:28:45.198866 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:45.198835    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerID="8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810" exitCode=0
Sep 18 18:28:45.199302 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:45.198905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" event={"ID":"4ab1e956-1d53-46ec-8b80-d40a22cff42d","Type":"ContainerDied","Data":"8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"}
Sep 18 18:28:45.199345 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:45.199328    2580 scope.go:117] "RemoveContainer" containerID="8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"
Sep 18 18:28:45.200354 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:45.200326    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:45.200408 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:45.200392    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:45.200443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:45.200435    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 18:28:46.206169 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:46.206131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" event={"ID":"4ab1e956-1d53-46ec-8b80-d40a22cff42d","Type":"ContainerStarted","Data":"987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976"}
Sep 18 18:28:47.106134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:47.106101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:47.106362 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:47.106337    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 18 18:28:47.106458 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:47.106404    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 18 18:28:48.105981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:48.105943    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 18 18:28:48.106504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:48.106011    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 18 18:28:49.106007 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:49.105968    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 18 18:28:49.106368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:49.106053    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 18 18:28:50.106455 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:50.106416    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 18 18:28:50.106830 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:50.106481    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 18 18:28:51.105849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:51.105812    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 18 18:28:51.106099 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:51.105866    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 18 18:28:52.107276 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:52.107238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:52.110692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:52.110669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:52.110965 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:52.110946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:52.111610 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:52.111596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:28:52.212357 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:52.212322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:52.820689 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:52.820653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:53.501303 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:53.501273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:54.159949 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:54.159888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:54.914289 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:54.914241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:28:57.348711 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:28:57.348677    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4_52d48752-1d52-4836-a475-a7238c9791b6/payload-processing/0.log"
Sep 18 18:29:04.309392 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:04.309346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:28.774850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:28.774796    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4_52d48752-1d52-4836-a475-a7238c9791b6/payload-processing/0.log"
Sep 18 18:29:29.039091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:29.039001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:29.345261 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:29.345173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:29.660573 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:29.660509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:32.000564 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:32.000534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:32.550784 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:32.550744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:32.732469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:32.732424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:32.831410 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:32.831329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:33.372575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:33.372532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:34.087767 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:34.087708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:34.243376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:34.243339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:34.342971 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:34.342875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:34.917522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:34.917489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:35.601664 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:35.601628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:35.674603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:35.674567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:35.737615 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:35.737577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:36.190243 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.190206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:36.283159 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.283127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:36.314337 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.314310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"]
Sep 18 18:29:36.317892 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.317872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.320065 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.320020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-12b560\""
Sep 18 18:29:36.320180 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.320164    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-12b560-dockercfg-d5wz7\""
Sep 18 18:29:36.325728 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.325701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"]
Sep 18 18:29:36.510736 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.510653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6hp\" (UniqueName: \"kubernetes.io/projected/349de999-c77b-4dbc-9ce8-3b8bd6415457-kube-api-access-cz6hp\") pod \"payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.510883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.510754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349de999-c77b-4dbc-9ce8-3b8bd6415457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.612037 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.611992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6hp\" (UniqueName: \"kubernetes.io/projected/349de999-c77b-4dbc-9ce8-3b8bd6415457-kube-api-access-cz6hp\") pod \"payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.612190 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.612067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349de999-c77b-4dbc-9ce8-3b8bd6415457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.612756 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.612733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349de999-c77b-4dbc-9ce8-3b8bd6415457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.619665 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.619643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6hp\" (UniqueName: \"kubernetes.io/projected/349de999-c77b-4dbc-9ce8-3b8bd6415457-kube-api-access-cz6hp\") pod \"payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.633513 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.633481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:36.904653 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:36.904625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"]
Sep 18 18:29:36.908148 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:29:36.908115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349de999_c77b_4dbc_9ce8_3b8bd6415457.slice/crio-4771ddfdc8939b31dc3f2375fb3c31c94b67391c2be2a61494ffbb6689920c63 WatchSource:0}: Error finding container 4771ddfdc8939b31dc3f2375fb3c31c94b67391c2be2a61494ffbb6689920c63: Status 404 returned error can't find the container with id 4771ddfdc8939b31dc3f2375fb3c31c94b67391c2be2a61494ffbb6689920c63
Sep 18 18:29:37.473046 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:37.472990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79" event={"ID":"349de999-c77b-4dbc-9ce8-3b8bd6415457","Type":"ContainerStarted","Data":"5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349"}
Sep 18 18:29:37.473447 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:37.473099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79" event={"ID":"349de999-c77b-4dbc-9ce8-3b8bd6415457","Type":"ContainerStarted","Data":"4771ddfdc8939b31dc3f2375fb3c31c94b67391c2be2a61494ffbb6689920c63"}
Sep 18 18:29:37.473447 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:37.473125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:37.487224 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:37.487172    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79" podStartSLOduration=1.487154853 podStartE2EDuration="1.487154853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:29:37.486307995 +0000 UTC m=+1747.638182830" watchObservedRunningTime="2026-09-18 18:29:37.487154853 +0000 UTC m=+1747.639029689"
Sep 18 18:29:47.464018 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:47.463980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:48.481479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:48.481448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:29:57.667316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:57.667269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:57.768638 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:57.768603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:58.346499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:58.346464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:58.878832 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:58.878798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:59.682682 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:29:59.682638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:00.136895 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.136817    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"]
Sep 18 18:30:00.140301 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.140282    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"]
Sep 18 18:30:00.140412 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.140392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"
Sep 18 18:30:00.142479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.142457    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4ghvf\""
Sep 18 18:30:00.143628 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.143610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"
Sep 18 18:30:00.148688 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.148660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"]
Sep 18 18:30:00.152882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.152859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"
Sep 18 18:30:00.153117 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.153100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"]
Sep 18 18:30:00.163017 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.162998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"
Sep 18 18:30:00.404860 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.404750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:00.432110 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.432084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"]
Sep 18 18:30:00.433603 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:30:00.433571    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da69f2e_4a95_4ebe_a1fe_bd3344d9777f.slice/crio-e60569f393ec5cb1d5b4887cf8eabae18c2f30ed4de0d424a4df978f63ccf084 WatchSource:0}: Error finding container e60569f393ec5cb1d5b4887cf8eabae18c2f30ed4de0d424a4df978f63ccf084: Status 404 returned error can't find the container with id e60569f393ec5cb1d5b4887cf8eabae18c2f30ed4de0d424a4df978f63ccf084
Sep 18 18:30:00.435894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.435876    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:30:00.446594 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.446545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"]
Sep 18 18:30:00.450633 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:30:00.450598    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4787314e_303d_47f5_92a7_e3671c7b2864.slice/crio-e660f1049f1b0b2994b45bcc4096b9d658b70d037e68cafabf714f83b047e0ca WatchSource:0}: Error finding container e660f1049f1b0b2994b45bcc4096b9d658b70d037e68cafabf714f83b047e0ca: Status 404 returned error can't find the container with id e660f1049f1b0b2994b45bcc4096b9d658b70d037e68cafabf714f83b047e0ca
Sep 18 18:30:00.590082 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.590045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r" event={"ID":"4787314e-303d-47f5-92a7-e3671c7b2864","Type":"ContainerStarted","Data":"e660f1049f1b0b2994b45bcc4096b9d658b70d037e68cafabf714f83b047e0ca"}
Sep 18 18:30:00.591249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:00.591219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv" event={"ID":"8da69f2e-4a95-4ebe-a1fe-bd3344d9777f","Type":"ContainerStarted","Data":"e60569f393ec5cb1d5b4887cf8eabae18c2f30ed4de0d424a4df978f63ccf084"}
Sep 18 18:30:01.081264 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:01.081232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:01.815009 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:01.814964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:02.594053 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:02.593979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:03.608692 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:03.608657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r" event={"ID":"4787314e-303d-47f5-92a7-e3671c7b2864","Type":"ContainerStarted","Data":"0a4b5ea4d1a2927fcca1ac414ac3219027e3ea3e70ef02710f17ac1ec9f287d0"}
Sep 18 18:30:03.610364 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:03.610340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv" event={"ID":"8da69f2e-4a95-4ebe-a1fe-bd3344d9777f","Type":"ContainerStarted","Data":"cf7bfdfe5e6032fcaf1b8bb30d124f521472eacc19cbcbd3fb5d4a4f8d992cf1"}
Sep 18 18:30:03.623352 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:03.623314    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r" podStartSLOduration=0.842516948 podStartE2EDuration="3.623301513s" totalImagesPullingTime="2.780784565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:30:03.621928949 +0000 UTC m=+1773.773803783" watchObservedRunningTime="2026-09-18 18:30:03.623301513 +0000 UTC m=+1773.775176368"
Sep 18 18:30:03.638205 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:03.638164    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv" podStartSLOduration=0.806520467 podStartE2EDuration="3.638151844s" totalImagesPullingTime="2.831631377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:30:03.638145203 +0000 UTC m=+1773.790020043" watchObservedRunningTime="2026-09-18 18:30:03.638151844 +0000 UTC m=+1773.790026729"
Sep 18 18:30:04.153797 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.153718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:04.616817 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.616782    2580 generic.go:356] "Generic (PLEG): container finished" podID="4787314e-303d-47f5-92a7-e3671c7b2864" containerID="0a4b5ea4d1a2927fcca1ac414ac3219027e3ea3e70ef02710f17ac1ec9f287d0" exitCode=0
Sep 18 18:30:04.617280 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.616876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r" event={"ID":"4787314e-303d-47f5-92a7-e3671c7b2864","Type":"ContainerDied","Data":"0a4b5ea4d1a2927fcca1ac414ac3219027e3ea3e70ef02710f17ac1ec9f287d0"}
Sep 18 18:30:04.618373 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.618353    2580 generic.go:356] "Generic (PLEG): container finished" podID="8da69f2e-4a95-4ebe-a1fe-bd3344d9777f" containerID="cf7bfdfe5e6032fcaf1b8bb30d124f521472eacc19cbcbd3fb5d4a4f8d992cf1" exitCode=0
Sep 18 18:30:04.618469 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.618445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv" event={"ID":"8da69f2e-4a95-4ebe-a1fe-bd3344d9777f","Type":"ContainerDied","Data":"cf7bfdfe5e6032fcaf1b8bb30d124f521472eacc19cbcbd3fb5d4a4f8d992cf1"}
Sep 18 18:30:04.689490 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.689448    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"]
Sep 18 18:30:04.705534 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.705501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:04.727400 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.727368    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"]
Sep 18 18:30:04.727913 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.727881    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79" podUID="349de999-c77b-4dbc-9ce8-3b8bd6415457" containerName="payload-processing" containerID="cri-o://5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349" gracePeriod=30
Sep 18 18:30:04.800520 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.800482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:04.901707 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.901596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:04.985910 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:04.985879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:30:05.004133 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.003972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:05.068883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.068847    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349de999-c77b-4dbc-9ce8-3b8bd6415457-plugins-config-volume\") pod \"349de999-c77b-4dbc-9ce8-3b8bd6415457\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") "
Sep 18 18:30:05.069116 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.068894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz6hp\" (UniqueName: \"kubernetes.io/projected/349de999-c77b-4dbc-9ce8-3b8bd6415457-kube-api-access-cz6hp\") pod \"349de999-c77b-4dbc-9ce8-3b8bd6415457\" (UID: \"349de999-c77b-4dbc-9ce8-3b8bd6415457\") "
Sep 18 18:30:05.069372 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.069344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349de999-c77b-4dbc-9ce8-3b8bd6415457-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "349de999-c77b-4dbc-9ce8-3b8bd6415457" (UID: "349de999-c77b-4dbc-9ce8-3b8bd6415457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:30:05.072351 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.072325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349de999-c77b-4dbc-9ce8-3b8bd6415457-kube-api-access-cz6hp" (OuterVolumeSpecName: "kube-api-access-cz6hp") pod "349de999-c77b-4dbc-9ce8-3b8bd6415457" (UID: "349de999-c77b-4dbc-9ce8-3b8bd6415457"). InnerVolumeSpecName "kube-api-access-cz6hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:30:05.170134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.170104    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349de999-c77b-4dbc-9ce8-3b8bd6415457-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:30:05.170134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.170130    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz6hp\" (UniqueName: \"kubernetes.io/projected/349de999-c77b-4dbc-9ce8-3b8bd6415457-kube-api-access-cz6hp\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:30:05.323403 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.323361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:05.424652 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.424560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:05.624564 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.624530    2580 generic.go:356] "Generic (PLEG): container finished" podID="349de999-c77b-4dbc-9ce8-3b8bd6415457" containerID="5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349" exitCode=0
Sep 18 18:30:05.625096 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.624585    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"
Sep 18 18:30:05.625096 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.624612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79" event={"ID":"349de999-c77b-4dbc-9ce8-3b8bd6415457","Type":"ContainerDied","Data":"5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349"}
Sep 18 18:30:05.625096 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.624696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79" event={"ID":"349de999-c77b-4dbc-9ce8-3b8bd6415457","Type":"ContainerDied","Data":"4771ddfdc8939b31dc3f2375fb3c31c94b67391c2be2a61494ffbb6689920c63"}
Sep 18 18:30:05.625096 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.624720    2580 scope.go:117] "RemoveContainer" containerID="5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349"
Sep 18 18:30:05.637642 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.637620    2580 scope.go:117] "RemoveContainer" containerID="5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349"
Sep 18 18:30:05.637972 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:30:05.637952    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349\": container with ID starting with 5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349 not found: ID does not exist" containerID="5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349"
Sep 18 18:30:05.638097 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.637980    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349"} err="failed to get container status \"5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349\": rpc error: code = NotFound desc = could not find container \"5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349\": container with ID starting with 5e39ec959c73e25747e5d2b435c2c1788d81951d703bfbdcff97a0f25e382349 not found: ID does not exist"
Sep 18 18:30:05.651409 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.651375    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"]
Sep 18 18:30:05.656522 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.655955    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-12b560-6876ddc695-p6w79"]
Sep 18 18:30:05.784788 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.784768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"
Sep 18 18:30:05.786844 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.786827    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"
Sep 18 18:30:05.910342 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:05.910308    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="349de999-c77b-4dbc-9ce8-3b8bd6415457" path="/var/lib/kubelet/pods/349de999-c77b-4dbc-9ce8-3b8bd6415457/volumes"
Sep 18 18:30:06.118450 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.118366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:06.631342 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.631308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r"
Sep 18 18:30:06.631786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.631344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829270-mts8r" event={"ID":"4787314e-303d-47f5-92a7-e3671c7b2864","Type":"ContainerDied","Data":"e660f1049f1b0b2994b45bcc4096b9d658b70d037e68cafabf714f83b047e0ca"}
Sep 18 18:30:06.631786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.631402    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e660f1049f1b0b2994b45bcc4096b9d658b70d037e68cafabf714f83b047e0ca"
Sep 18 18:30:06.633868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.633844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv" event={"ID":"8da69f2e-4a95-4ebe-a1fe-bd3344d9777f","Type":"ContainerDied","Data":"e60569f393ec5cb1d5b4887cf8eabae18c2f30ed4de0d424a4df978f63ccf084"}
Sep 18 18:30:06.633868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.633880    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60569f393ec5cb1d5b4887cf8eabae18c2f30ed4de0d424a4df978f63ccf084"
Sep 18 18:30:06.634063 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.634042    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"
Sep 18 18:30:06.658838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.658810    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"]
Sep 18 18:30:06.663983 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.663960    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-12b560-29829270-ktnbv"]
Sep 18 18:30:06.787307 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:06.787273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:07.465649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:07.465604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:07.909560 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:07.909536    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da69f2e-4a95-4ebe-a1fe-bd3344d9777f" path="/var/lib/kubelet/pods/8da69f2e-4a95-4ebe-a1fe-bd3344d9777f/volumes"
Sep 18 18:30:08.111174 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:08.111136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:11.149279 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:11.149244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:11.204716 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:11.204687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:11.415870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:11.415837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:25.600639 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:25.600602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:26.403846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:26.403807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:27.211525 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:27.211489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:27.836575 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:27.836535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:28.595393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:28.595348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:29.229042 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:29.229006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:40.920733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:40.920691    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"]
Sep 18 18:30:40.921262 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:40.920954    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5" podUID="53965664-7b53-42e6-a85d-da75cc58acd8" containerName="maas-api" containerID="cri-o://cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202" gracePeriod=30
Sep 18 18:30:40.962275 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:40.962230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:41.167371 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.167343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:30:41.297720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.297637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-metrics-certs\") pod \"53965664-7b53-42e6-a85d-da75cc58acd8\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") "
Sep 18 18:30:41.297720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.297686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53965664-7b53-42e6-a85d-da75cc58acd8-openshift-service-ca\") pod \"53965664-7b53-42e6-a85d-da75cc58acd8\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") "
Sep 18 18:30:41.297945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.297724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gpmb\" (UniqueName: \"kubernetes.io/projected/53965664-7b53-42e6-a85d-da75cc58acd8-kube-api-access-4gpmb\") pod \"53965664-7b53-42e6-a85d-da75cc58acd8\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") "
Sep 18 18:30:41.297945 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.297792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-maas-api-tls\") pod \"53965664-7b53-42e6-a85d-da75cc58acd8\" (UID: \"53965664-7b53-42e6-a85d-da75cc58acd8\") "
Sep 18 18:30:41.298140 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.298105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53965664-7b53-42e6-a85d-da75cc58acd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53965664-7b53-42e6-a85d-da75cc58acd8" (UID: "53965664-7b53-42e6-a85d-da75cc58acd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:30:41.300124 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.300016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53965664-7b53-42e6-a85d-da75cc58acd8-kube-api-access-4gpmb" (OuterVolumeSpecName: "kube-api-access-4gpmb") pod "53965664-7b53-42e6-a85d-da75cc58acd8" (UID: "53965664-7b53-42e6-a85d-da75cc58acd8"). InnerVolumeSpecName "kube-api-access-4gpmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:30:41.300124 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.300088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53965664-7b53-42e6-a85d-da75cc58acd8" (UID: "53965664-7b53-42e6-a85d-da75cc58acd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:30:41.300247 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.300182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53965664-7b53-42e6-a85d-da75cc58acd8" (UID: "53965664-7b53-42e6-a85d-da75cc58acd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:30:41.399411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.399380    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53965664-7b53-42e6-a85d-da75cc58acd8-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:30:41.399411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.399406    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpmb\" (UniqueName: \"kubernetes.io/projected/53965664-7b53-42e6-a85d-da75cc58acd8-kube-api-access-4gpmb\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:30:41.399411 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.399415    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:30:41.399616 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.399424    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53965664-7b53-42e6-a85d-da75cc58acd8-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:30:41.821398 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.821367    2580 generic.go:356] "Generic (PLEG): container finished" podID="53965664-7b53-42e6-a85d-da75cc58acd8" containerID="cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202" exitCode=0
Sep 18 18:30:41.821577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.821404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5" event={"ID":"53965664-7b53-42e6-a85d-da75cc58acd8","Type":"ContainerDied","Data":"cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202"}
Sep 18 18:30:41.821577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.821412    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"
Sep 18 18:30:41.821577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.821447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5" event={"ID":"53965664-7b53-42e6-a85d-da75cc58acd8","Type":"ContainerDied","Data":"298ddb33dbef805b025f4a5a5ae9169f5536097d63b23c5553af1f34479a90f2"}
Sep 18 18:30:41.821577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.821467    2580 scope.go:117] "RemoveContainer" containerID="cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202"
Sep 18 18:30:41.833108 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.833084    2580 scope.go:117] "RemoveContainer" containerID="cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202"
Sep 18 18:30:41.833391 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:30:41.833373    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202\": container with ID starting with cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202 not found: ID does not exist" containerID="cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202"
Sep 18 18:30:41.833443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.833398    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202"} err="failed to get container status \"cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202\": rpc error: code = NotFound desc = could not find container \"cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202\": container with ID starting with cd8b9377e904c3a736fd47a11ff7cd80bd2a3980df8092ad3665b0efb68f4202 not found: ID does not exist"
Sep 18 18:30:41.845203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.845178    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"]
Sep 18 18:30:41.849163 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.849141    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e8cc-76d896fb58-vfsk5"]
Sep 18 18:30:41.910057 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:41.910011    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53965664-7b53-42e6-a85d-da75cc58acd8" path="/var/lib/kubelet/pods/53965664-7b53-42e6-a85d-da75cc58acd8/volumes"
Sep 18 18:30:42.413138 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:42.413107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:43.024529 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:43.024494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:43.655447 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:43.655410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:44.396439 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:44.396402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:46.666478 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:46.666440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:47.199078 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:47.199015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:47.266166 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:47.266131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:58.268894 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:58.268850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:58.319334 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:58.319286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:59.065820 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:59.065777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:30:59.118889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:30:59.118854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:00.255728 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:00.255699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:00.356364 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:00.356334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:12.278335 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:12.278296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:22.757760 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:22.757721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:23.647369 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:23.647328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:26.948188 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:26.948152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:27.049937 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:27.049902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:29.101066 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:29.100988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:29.162380 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:29.162340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:29.401574 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:29.401491    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"]
Sep 18 18:31:29.401835 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:29.401801    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy" containerID="cri-o://a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892" gracePeriod=30
Sep 18 18:31:29.421767 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:29.421736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:30.115408 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:30.115343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:30.926649 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:30.926607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:31.562846 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:31.562804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:32.209249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:32.209222    2580 scope.go:117] "RemoveContainer" containerID="7fa9d820796a40872239d0aad9fa0bde559de721e9e80bf89ed72f9a04b49a93"
Sep 18 18:31:32.271350 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:32.271321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:32.923454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:32.923422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:34.646943 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.646915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:31:34.794873 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.794800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-envoy\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.794873 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.794860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-certs\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.794897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfksw\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-kube-api-access-qfksw\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.794921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-data\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.794956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-credential-socket\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.794983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-podinfo\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee0eb77-299e-4050-b23b-7ef359bc26bb-istiod-ca-cert\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-token\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-socket\") pod \"eee0eb77-299e-4050-b23b-7ef359bc26bb\" (UID: \"eee0eb77-299e-4050-b23b-7ef359bc26bb\") "
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795252    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795534    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795555    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.795563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795567    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.795870 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.795670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee0eb77-299e-4050-b23b-7ef359bc26bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:31:34.797373 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.797343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:31:34.797373 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.797355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-kube-api-access-qfksw" (OuterVolumeSpecName: "kube-api-access-qfksw") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "kube-api-access-qfksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:31:34.797723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.797700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:31:34.797723 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.797706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eee0eb77-299e-4050-b23b-7ef359bc26bb" (UID: "eee0eb77-299e-4050-b23b-7ef359bc26bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:34.895981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.895956    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.895981 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.895977    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.896149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.895986    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfksw\" (UniqueName: \"kubernetes.io/projected/eee0eb77-299e-4050-b23b-7ef359bc26bb-kube-api-access-qfksw\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.896149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.895995    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee0eb77-299e-4050-b23b-7ef359bc26bb-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.896149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.896003    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee0eb77-299e-4050-b23b-7ef359bc26bb-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:34.896149 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:34.896011    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee0eb77-299e-4050-b23b-7ef359bc26bb-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:35.085213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.085136    2580 generic.go:356] "Generic (PLEG): container finished" podID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerID="a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892" exitCode=0
Sep 18 18:31:35.085213 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.085188    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"
Sep 18 18:31:35.085383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.085227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" event={"ID":"eee0eb77-299e-4050-b23b-7ef359bc26bb","Type":"ContainerDied","Data":"a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892"}
Sep 18 18:31:35.085383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.085290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh" event={"ID":"eee0eb77-299e-4050-b23b-7ef359bc26bb","Type":"ContainerDied","Data":"d0ba093d77975a7332440449533aa733439847f92753b8b6a4e1ad0775b3b725"}
Sep 18 18:31:35.085383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.085314    2580 scope.go:117] "RemoveContainer" containerID="a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892"
Sep 18 18:31:35.094919 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.094901    2580 scope.go:117] "RemoveContainer" containerID="2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"
Sep 18 18:31:35.102922 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.102907    2580 scope.go:117] "RemoveContainer" containerID="a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892"
Sep 18 18:31:35.103219 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:31:35.103190    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892\": container with ID starting with a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892 not found: ID does not exist" containerID="a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892"
Sep 18 18:31:35.103308 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.103227    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892"} err="failed to get container status \"a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892\": rpc error: code = NotFound desc = could not find container \"a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892\": container with ID starting with a13369448d9754fc705403f75c3e33ca62746a8327fcc45a41c49d7335438892 not found: ID does not exist"
Sep 18 18:31:35.103308 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.103249    2580 scope.go:117] "RemoveContainer" containerID="2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"
Sep 18 18:31:35.103545 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:31:35.103524    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7\": container with ID starting with 2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7 not found: ID does not exist" containerID="2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"
Sep 18 18:31:35.103589 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.103551    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7"} err="failed to get container status \"2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7\": rpc error: code = NotFound desc = could not find container \"2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7\": container with ID starting with 2520920191fda6921b6d554fb694295ec5f8e041e72d83c006161255a9ed82f7 not found: ID does not exist"
Sep 18 18:31:35.106776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.106752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"]
Sep 18 18:31:35.110850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.110829    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e8cc-openshift-default-6b665647bd-z9kmh"]
Sep 18 18:31:35.910900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:35.910865    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" path="/var/lib/kubelet/pods/eee0eb77-299e-4050-b23b-7ef359bc26bb/volumes"
Sep 18 18:31:40.308467 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.308436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:40.336884 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.336851    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"]
Sep 18 18:31:40.337195 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.337164    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4" podUID="52d48752-1d52-4836-a475-a7238c9791b6" containerName="payload-processing" containerID="cri-o://b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6" gracePeriod=30
Sep 18 18:31:40.377429 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.377399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:40.477867 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.477801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:40.532787 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.532749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:40.594897 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.594871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:31:40.747225 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.747190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d48752-1d52-4836-a475-a7238c9791b6-plugins-config-volume\") pod \"52d48752-1d52-4836-a475-a7238c9791b6\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") "
Sep 18 18:31:40.747225 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.747233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqjz6\" (UniqueName: \"kubernetes.io/projected/52d48752-1d52-4836-a475-a7238c9791b6-kube-api-access-pqjz6\") pod \"52d48752-1d52-4836-a475-a7238c9791b6\" (UID: \"52d48752-1d52-4836-a475-a7238c9791b6\") "
Sep 18 18:31:40.747597 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.747573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d48752-1d52-4836-a475-a7238c9791b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52d48752-1d52-4836-a475-a7238c9791b6" (UID: "52d48752-1d52-4836-a475-a7238c9791b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:31:40.749472 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.749451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d48752-1d52-4836-a475-a7238c9791b6-kube-api-access-pqjz6" (OuterVolumeSpecName: "kube-api-access-pqjz6") pod "52d48752-1d52-4836-a475-a7238c9791b6" (UID: "52d48752-1d52-4836-a475-a7238c9791b6"). InnerVolumeSpecName "kube-api-access-pqjz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:31:40.848368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.848335    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d48752-1d52-4836-a475-a7238c9791b6-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:40.848368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.848363    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjz6\" (UniqueName: \"kubernetes.io/projected/52d48752-1d52-4836-a475-a7238c9791b6-kube-api-access-pqjz6\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:40.915349 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:40.915317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:41.015577 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.015498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:41.116160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.116128    2580 generic.go:356] "Generic (PLEG): container finished" podID="52d48752-1d52-4836-a475-a7238c9791b6" containerID="b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6" exitCode=0
Sep 18 18:31:41.116329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.116171    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"
Sep 18 18:31:41.116329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.116219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4" event={"ID":"52d48752-1d52-4836-a475-a7238c9791b6","Type":"ContainerDied","Data":"b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6"}
Sep 18 18:31:41.116329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.116265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4" event={"ID":"52d48752-1d52-4836-a475-a7238c9791b6","Type":"ContainerDied","Data":"eeb855df5f9ad73894e33a32f899dce3a2c9bce4900e04427ab15c812ac6a24d"}
Sep 18 18:31:41.116329 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.116296    2580 scope.go:117] "RemoveContainer" containerID="b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6"
Sep 18 18:31:41.125743 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.125720    2580 scope.go:117] "RemoveContainer" containerID="b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6"
Sep 18 18:31:41.126207 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:31:41.126172    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6\": container with ID starting with b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6 not found: ID does not exist" containerID="b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6"
Sep 18 18:31:41.126288 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.126216    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6"} err="failed to get container status \"b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6\": rpc error: code = NotFound desc = could not find container \"b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6\": container with ID starting with b52e2509c3789696529d51da221a5f3dfbb1b2705036f204aa3a955ab89c7cc6 not found: ID does not exist"
Sep 18 18:31:41.140216 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.140197    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"]
Sep 18 18:31:41.144089 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.144066    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46c63c-6866967f69-2mnf4"]
Sep 18 18:31:41.682275 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.682237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:41.910290 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:41.910261    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d48752-1d52-4836-a475-a7238c9791b6" path="/var/lib/kubelet/pods/52d48752-1d52-4836-a475-a7238c9791b6/volumes"
Sep 18 18:31:42.321102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:42.321003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:46.833985 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:46.833938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:46.895295 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:46.895265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:47.151939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:47.151847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"]
Sep 18 18:31:47.152162 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:47.152132    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" containerID="cri-o://987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976" gracePeriod=30
Sep 18 18:31:47.221734 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:47.221684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:47.621440 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:47.621402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:48.408135 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:48.408098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:49.095598 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:49.095554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:49.819286 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:49.819252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:52.113360 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.113328    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 18:31:52.113760 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.113401    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 18:31:52.405094 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.405069    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:31:52.451095 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451066    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-podinfo\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-certs\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451126    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-token\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-data\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-envoy\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-socket\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8b84\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-kube-api-access-d8b84\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451260 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-credential-socket\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istiod-ca-cert\") pod \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\" (UID: \"4ab1e956-1d53-46ec-8b80-d40a22cff42d\") "
Sep 18 18:31:52.451631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:52.451631 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451609    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.451799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451677    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:52.451799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:31:52.451799 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:52.452041 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.451969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:52.453444 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.453408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:31:52.453570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.453440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:31:52.453570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.453520    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:31:52.453657 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.453592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-kube-api-access-d8b84" (OuterVolumeSpecName: "kube-api-access-d8b84") pod "4ab1e956-1d53-46ec-8b80-d40a22cff42d" (UID: "4ab1e956-1d53-46ec-8b80-d40a22cff42d"). InnerVolumeSpecName "kube-api-access-d8b84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:31:52.552284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552253    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552279    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552284 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552287    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552509 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552295    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552509 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552304    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552509 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552312    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8b84\" (UniqueName: \"kubernetes.io/projected/4ab1e956-1d53-46ec-8b80-d40a22cff42d-kube-api-access-d8b84\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552509 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552320    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1e956-1d53-46ec-8b80-d40a22cff42d-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:52.552509 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:52.552328    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1e956-1d53-46ec-8b80-d40a22cff42d-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:53.175678 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.175640    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerID="987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976" exitCode=0
Sep 18 18:31:53.176072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.175692    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"
Sep 18 18:31:53.176072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.175705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" event={"ID":"4ab1e956-1d53-46ec-8b80-d40a22cff42d","Type":"ContainerDied","Data":"987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976"}
Sep 18 18:31:53.176072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.175759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq" event={"ID":"4ab1e956-1d53-46ec-8b80-d40a22cff42d","Type":"ContainerDied","Data":"152fdd04e638143886c67f0918b506d1c6efe2e9f0c911c2e28890379f6f336c"}
Sep 18 18:31:53.176072 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.175787    2580 scope.go:117] "RemoveContainer" containerID="987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976"
Sep 18 18:31:53.185315 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.185294    2580 scope.go:117] "RemoveContainer" containerID="8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"
Sep 18 18:31:53.193082 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.193066    2580 scope.go:117] "RemoveContainer" containerID="987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976"
Sep 18 18:31:53.193358 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:31:53.193340    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976\": container with ID starting with 987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976 not found: ID does not exist" containerID="987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976"
Sep 18 18:31:53.193408 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.193368    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976"} err="failed to get container status \"987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976\": rpc error: code = NotFound desc = could not find container \"987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976\": container with ID starting with 987612f4d85dcd26fb35507ff705d310ad5bf5974d3180337a5a27a0e1a07976 not found: ID does not exist"
Sep 18 18:31:53.193408 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.193385    2580 scope.go:117] "RemoveContainer" containerID="8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"
Sep 18 18:31:53.193627 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:31:53.193608    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810\": container with ID starting with 8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810 not found: ID does not exist" containerID="8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"
Sep 18 18:31:53.193673 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.193635    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810"} err="failed to get container status \"8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810\": rpc error: code = NotFound desc = could not find container \"8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810\": container with ID starting with 8b781d554a49435f2690ccb618e369cf9987d81ebb98eb4759540e12e25e6810 not found: ID does not exist"
Sep 18 18:31:53.196501 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.196481    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"]
Sep 18 18:31:53.199746 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.199726    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-46c63c-openshift-default-bdc9c597d-cj4sq"]
Sep 18 18:31:53.913088 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:53.910916    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" path="/var/lib/kubelet/pods/4ab1e956-1d53-46ec-8b80-d40a22cff42d/volumes"
Sep 18 18:31:57.969588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:57.969556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:57.993853 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:57.993815    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"]
Sep 18 18:31:57.994124 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:57.994094    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb" podUID="be554f45-681e-4bea-bfa5-06ea1169936f" containerName="payload-processing" containerID="cri-o://60bcb86fd0731bbca224f770500e6b5119c3728ddf4eb5dde62248bac13b7193" gracePeriod=30
Sep 18 18:31:58.069688 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.069649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:58.169504 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.169450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:58.200325 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.200292    2580 generic.go:356] "Generic (PLEG): container finished" podID="be554f45-681e-4bea-bfa5-06ea1169936f" containerID="60bcb86fd0731bbca224f770500e6b5119c3728ddf4eb5dde62248bac13b7193" exitCode=0
Sep 18 18:31:58.200482 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.200368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb" event={"ID":"be554f45-681e-4bea-bfa5-06ea1169936f","Type":"ContainerDied","Data":"60bcb86fd0731bbca224f770500e6b5119c3728ddf4eb5dde62248bac13b7193"}
Sep 18 18:31:58.260075 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.260049    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:31:58.267385 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.267335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:58.304261 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.304229    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be554f45-681e-4bea-bfa5-06ea1169936f-plugins-config-volume\") pod \"be554f45-681e-4bea-bfa5-06ea1169936f\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") "
Sep 18 18:31:58.304406 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.304337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdmms\" (UniqueName: \"kubernetes.io/projected/be554f45-681e-4bea-bfa5-06ea1169936f-kube-api-access-cdmms\") pod \"be554f45-681e-4bea-bfa5-06ea1169936f\" (UID: \"be554f45-681e-4bea-bfa5-06ea1169936f\") "
Sep 18 18:31:58.304585 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.304564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be554f45-681e-4bea-bfa5-06ea1169936f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be554f45-681e-4bea-bfa5-06ea1169936f" (UID: "be554f45-681e-4bea-bfa5-06ea1169936f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:31:58.306491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.306472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be554f45-681e-4bea-bfa5-06ea1169936f-kube-api-access-cdmms" (OuterVolumeSpecName: "kube-api-access-cdmms") pod "be554f45-681e-4bea-bfa5-06ea1169936f" (UID: "be554f45-681e-4bea-bfa5-06ea1169936f"). InnerVolumeSpecName "kube-api-access-cdmms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:31:58.405125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.405074    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be554f45-681e-4bea-bfa5-06ea1169936f-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:58.405125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.405116    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdmms\" (UniqueName: \"kubernetes.io/projected/be554f45-681e-4bea-bfa5-06ea1169936f-kube-api-access-cdmms\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:31:58.588733 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.588623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:58.665632 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:58.665600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:59.206185 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.206154    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"
Sep 18 18:31:59.206185 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.206182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb" event={"ID":"be554f45-681e-4bea-bfa5-06ea1169936f","Type":"ContainerDied","Data":"9069f2f82aa73501d3024145d2fa8af4c38b28cf6508ff51dc6c231a1584d8c5"}
Sep 18 18:31:59.206655 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.206237    2580 scope.go:117] "RemoveContainer" containerID="60bcb86fd0731bbca224f770500e6b5119c3728ddf4eb5dde62248bac13b7193"
Sep 18 18:31:59.226479 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.226453    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"]
Sep 18 18:31:59.236043 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.236006    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-011541-865b6668df-c46rb"]
Sep 18 18:31:59.404742 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.404704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:59.909991 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:31:59.909951    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be554f45-681e-4bea-bfa5-06ea1169936f" path="/var/lib/kubelet/pods/be554f45-681e-4bea-bfa5-06ea1169936f/volumes"
Sep 18 18:32:02.217879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:02.217845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:04.372535 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:04.372492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:04.446624 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:04.446595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:04.632955 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:04.632875    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"]
Sep 18 18:32:04.633157 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:04.633133    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy" containerID="cri-o://171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6" gracePeriod=30
Sep 18 18:32:04.665184 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:04.665150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:05.464803 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:05.464768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:06.089645 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:06.089612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:06.818765 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:06.818734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:07.088349 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:07.088267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:07.186145 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:07.186110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:09.886059 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:09.886010    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:32:10.008636 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-podinfo\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.008789 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttpgk\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-kube-api-access-ttpgk\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.008789 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-data\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.008789 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-credential-socket\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.008789 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008777    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-certs\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.009010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-envoy\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.009010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008812    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a6bc8f-d788-45ad-93f0-1713657e3bac-istiod-ca-cert\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.009010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-socket\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.009010 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.008846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-token\") pod \"60a6bc8f-d788-45ad-93f0-1713657e3bac\" (UID: \"60a6bc8f-d788-45ad-93f0-1713657e3bac\") "
Sep 18 18:32:10.009273 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.009044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-data" (OuterVolumeSpecName: "istio-data") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:10.009273 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.009061    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:10.009273 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.009207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:10.009605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.009275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a6bc8f-d788-45ad-93f0-1713657e3bac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:32:10.009605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.009344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:10.011011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.010982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-kube-api-access-ttpgk" (OuterVolumeSpecName: "kube-api-access-ttpgk") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "kube-api-access-ttpgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:32:10.011011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.010982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:32:10.011011 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.011005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:10.011377 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.011357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-token" (OuterVolumeSpecName: "istio-token") pod "60a6bc8f-d788-45ad-93f0-1713657e3bac" (UID: "60a6bc8f-d788-45ad-93f0-1713657e3bac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:32:10.110420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110397    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110420 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110417    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110428    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110436    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110443    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a6bc8f-d788-45ad-93f0-1713657e3bac-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110450    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a6bc8f-d788-45ad-93f0-1713657e3bac-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110457    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110467    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a6bc8f-d788-45ad-93f0-1713657e3bac-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.110557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.110474    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttpgk\" (UniqueName: \"kubernetes.io/projected/60a6bc8f-d788-45ad-93f0-1713657e3bac-kube-api-access-ttpgk\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:10.269798 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.269728    2580 generic.go:356] "Generic (PLEG): container finished" podID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerID="171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6" exitCode=0
Sep 18 18:32:10.269798 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.269772    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"
Sep 18 18:32:10.269933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.269806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" event={"ID":"60a6bc8f-d788-45ad-93f0-1713657e3bac","Type":"ContainerDied","Data":"171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6"}
Sep 18 18:32:10.269933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.269863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h" event={"ID":"60a6bc8f-d788-45ad-93f0-1713657e3bac","Type":"ContainerDied","Data":"37f777138faa91df9376c10cb9e8d868d586e6963fe482fc5eef4f02ac2f6fa3"}
Sep 18 18:32:10.269933 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.269885    2580 scope.go:117] "RemoveContainer" containerID="171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6"
Sep 18 18:32:10.278881 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.278861    2580 scope.go:117] "RemoveContainer" containerID="03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"
Sep 18 18:32:10.287417 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.287386    2580 scope.go:117] "RemoveContainer" containerID="171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6"
Sep 18 18:32:10.287925 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:32:10.287894    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6\": container with ID starting with 171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6 not found: ID does not exist" containerID="171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6"
Sep 18 18:32:10.288062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.287934    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6"} err="failed to get container status \"171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6\": rpc error: code = NotFound desc = could not find container \"171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6\": container with ID starting with 171b8145c7018fd7942dbc6c4ec1cdb7f8676292ec35ed91477783b7174f0da6 not found: ID does not exist"
Sep 18 18:32:10.288062 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.287961    2580 scope.go:117] "RemoveContainer" containerID="03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"
Sep 18 18:32:10.288353 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:32:10.288293    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f\": container with ID starting with 03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f not found: ID does not exist" containerID="03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"
Sep 18 18:32:10.288353 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.288323    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f"} err="failed to get container status \"03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f\": rpc error: code = NotFound desc = could not find container \"03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f\": container with ID starting with 03286f18f4e74d1c3b8012a4959207720fc1ba8ba54ed5681c4518219e49cd6f not found: ID does not exist"
Sep 18 18:32:10.289588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.289569    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"]
Sep 18 18:32:10.293849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:10.293826    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-011541-openshift-default-cbd47d786-kxc7h"]
Sep 18 18:32:11.909890 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:11.909859    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" path="/var/lib/kubelet/pods/60a6bc8f-d788-45ad-93f0-1713657e3bac/volumes"
Sep 18 18:32:15.543071 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.543015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:15.571592 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.571560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"]
Sep 18 18:32:15.572106 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.572062    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv" podUID="21fc15ee-34ca-4f87-a743-c845accfa5d2" containerName="payload-pre-processing" containerID="cri-o://01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978" gracePeriod=30
Sep 18 18:32:15.642222 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.642168    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:15.742777 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.742741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:15.817831 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.817810    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:32:15.822343 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.822316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:15.955969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.955938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fc15ee-34ca-4f87-a743-c845accfa5d2-plugins-config-volume\") pod \"21fc15ee-34ca-4f87-a743-c845accfa5d2\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") "
Sep 18 18:32:15.956151 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.955984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjgbn\" (UniqueName: \"kubernetes.io/projected/21fc15ee-34ca-4f87-a743-c845accfa5d2-kube-api-access-zjgbn\") pod \"21fc15ee-34ca-4f87-a743-c845accfa5d2\" (UID: \"21fc15ee-34ca-4f87-a743-c845accfa5d2\") "
Sep 18 18:32:15.956375 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.956348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21fc15ee-34ca-4f87-a743-c845accfa5d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21fc15ee-34ca-4f87-a743-c845accfa5d2" (UID: "21fc15ee-34ca-4f87-a743-c845accfa5d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:32:15.958251 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:15.958232    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fc15ee-34ca-4f87-a743-c845accfa5d2-kube-api-access-zjgbn" (OuterVolumeSpecName: "kube-api-access-zjgbn") pod "21fc15ee-34ca-4f87-a743-c845accfa5d2" (UID: "21fc15ee-34ca-4f87-a743-c845accfa5d2"). InnerVolumeSpecName "kube-api-access-zjgbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:32:16.057003 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.056932    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fc15ee-34ca-4f87-a743-c845accfa5d2-plugins-config-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:16.057003 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.056967    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjgbn\" (UniqueName: \"kubernetes.io/projected/21fc15ee-34ca-4f87-a743-c845accfa5d2-kube-api-access-zjgbn\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:32:16.119678 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.119646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:16.220242 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.220207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:16.299749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.299720    2580 generic.go:356] "Generic (PLEG): container finished" podID="21fc15ee-34ca-4f87-a743-c845accfa5d2" containerID="01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978" exitCode=0
Sep 18 18:32:16.299893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.299767    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"
Sep 18 18:32:16.299893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.299775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv" event={"ID":"21fc15ee-34ca-4f87-a743-c845accfa5d2","Type":"ContainerDied","Data":"01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978"}
Sep 18 18:32:16.299893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.299844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv" event={"ID":"21fc15ee-34ca-4f87-a743-c845accfa5d2","Type":"ContainerDied","Data":"43d715579040244b1d153e29f183b0c6f196a81214783923ba34ef2bd1676e13"}
Sep 18 18:32:16.299893 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.299875    2580 scope.go:117] "RemoveContainer" containerID="01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978"
Sep 18 18:32:16.309673 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.309656    2580 scope.go:117] "RemoveContainer" containerID="01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978"
Sep 18 18:32:16.309936 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:32:16.309918    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978\": container with ID starting with 01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978 not found: ID does not exist" containerID="01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978"
Sep 18 18:32:16.310008 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.309947    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978"} err="failed to get container status \"01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978\": rpc error: code = NotFound desc = could not find container \"01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978\": container with ID starting with 01734f71e1411a2715f4282e78644ea52a8305627e1546bc4114f74ab576f978 not found: ID does not exist"
Sep 18 18:32:16.321020 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.320999    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"]
Sep 18 18:32:16.325362 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:16.325342    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99602c-66c978856b-kjttv"]
Sep 18 18:32:17.909831 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:17.909797    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21fc15ee-34ca-4f87-a743-c845accfa5d2" path="/var/lib/kubelet/pods/21fc15ee-34ca-4f87-a743-c845accfa5d2/volumes"
Sep 18 18:32:22.144558 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:22.144509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:22.245850 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:22.245816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:22.432107 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:32:22.432060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:33:26.701462 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701386    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-5v5x8"]
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701733    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701744    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701755    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21fc15ee-34ca-4f87-a743-c845accfa5d2" containerName="payload-pre-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701761    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21fc15ee-34ca-4f87-a743-c845accfa5d2" containerName="payload-pre-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701772    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349de999-c77b-4dbc-9ce8-3b8bd6415457" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701777    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="349de999-c77b-4dbc-9ce8-3b8bd6415457" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701784    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701789    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701795    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d48752-1d52-4836-a475-a7238c9791b6" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701800    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d48752-1d52-4836-a475-a7238c9791b6" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701807    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da69f2e-4a95-4ebe-a1fe-bd3344d9777f" containerName="cleanup"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701812    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da69f2e-4a95-4ebe-a1fe-bd3344d9777f" containerName="cleanup"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53965664-7b53-42e6-a85d-da75cc58acd8" containerName="maas-api"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701823    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53965664-7b53-42e6-a85d-da75cc58acd8" containerName="maas-api"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701829    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4787314e-303d-47f5-92a7-e3671c7b2864" containerName="cleanup"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701834    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4787314e-303d-47f5-92a7-e3671c7b2864" containerName="cleanup"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701840    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701845    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701852    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701857    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701867    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be554f45-681e-4bea-bfa5-06ea1169936f" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be554f45-681e-4bea-bfa5-06ea1169936f" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701929    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21fc15ee-34ca-4f87-a743-c845accfa5d2" containerName="payload-pre-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701937    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="349de999-c77b-4dbc-9ce8-3b8bd6415457" containerName="payload-processing"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701944    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53965664-7b53-42e6-a85d-da75cc58acd8" containerName="maas-api"
Sep 18 18:33:26.701939 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701952    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701957    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701963    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701967    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da69f2e-4a95-4ebe-a1fe-bd3344d9777f" containerName="cleanup"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701973    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701978    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be554f45-681e-4bea-bfa5-06ea1169936f" containerName="payload-processing"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701982    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a6bc8f-d788-45ad-93f0-1713657e3bac" containerName="istio-proxy"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701987    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d48752-1d52-4836-a475-a7238c9791b6" containerName="payload-processing"
Sep 18 18:33:26.702849 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.701992    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4787314e-303d-47f5-92a7-e3671c7b2864" containerName="cleanup"
Sep 18 18:33:26.704886 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.704870    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.707845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.707819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 18:33:26.707845 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.707842    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 18:33:26.708066 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.707855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 18:33:26.708066 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.707841    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xxjjv\""
Sep 18 18:33:26.713541 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.713520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-5v5x8"]
Sep 18 18:33:26.826219 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.826190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-metrics-certs\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.826376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.826223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-webhook-cert\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.826376 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.826257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpfmg\" (UniqueName: \"kubernetes.io/projected/f7c0aa61-6f0e-454e-9708-83413d640422-kube-api-access-jpfmg\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.927514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.927481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-metrics-certs\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.927514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.927513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-webhook-cert\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.927698 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.927540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpfmg\" (UniqueName: \"kubernetes.io/projected/f7c0aa61-6f0e-454e-9708-83413d640422-kube-api-access-jpfmg\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.930114 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.930087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-webhook-cert\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.930262 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.930242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-metrics-certs\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:26.935401 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:26.935379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpfmg\" (UniqueName: \"kubernetes.io/projected/f7c0aa61-6f0e-454e-9708-83413d640422-kube-api-access-jpfmg\") pod \"maas-controller-7dc7cc4dff-5v5x8\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") " pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:27.016217 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:27.016143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:27.286073 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:27.286044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-5v5x8"]
Sep 18 18:33:27.288165 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:33:27.288135    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c0aa61_6f0e_454e_9708_83413d640422.slice/crio-7ff8b6e720e6d5b0eebafb0a46583a1854c8efac2cfbcc533429673de3811c18 WatchSource:0}: Error finding container 7ff8b6e720e6d5b0eebafb0a46583a1854c8efac2cfbcc533429673de3811c18: Status 404 returned error can't find the container with id 7ff8b6e720e6d5b0eebafb0a46583a1854c8efac2cfbcc533429673de3811c18
Sep 18 18:33:27.625552 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:27.625460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8" event={"ID":"f7c0aa61-6f0e-454e-9708-83413d640422","Type":"ContainerStarted","Data":"7ff8b6e720e6d5b0eebafb0a46583a1854c8efac2cfbcc533429673de3811c18"}
Sep 18 18:33:28.631314 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:28.631274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8" event={"ID":"f7c0aa61-6f0e-454e-9708-83413d640422","Type":"ContainerStarted","Data":"4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b"}
Sep 18 18:33:28.631786 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:28.631404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:28.649241 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:28.649194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8" podStartSLOduration=1.984352772 podStartE2EDuration="2.649179959s" totalImagesPullingTime="664.827187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:33:28.647115751 +0000 UTC m=+1978.798990587" watchObservedRunningTime="2026-09-18 18:33:28.649179959 +0000 UTC m=+1978.801054793"
Sep 18 18:33:39.638102 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:39.638069    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:48.940776 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:48.940723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:33:49.040316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.040286    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-5v5x8"]
Sep 18 18:33:49.040499 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.040477    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8" podUID="f7c0aa61-6f0e-454e-9708-83413d640422" containerName="manager" containerID="cri-o://4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b" gracePeriod=30
Sep 18 18:33:49.286659 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.286635    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:49.415203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.415170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpfmg\" (UniqueName: \"kubernetes.io/projected/f7c0aa61-6f0e-454e-9708-83413d640422-kube-api-access-jpfmg\") pod \"f7c0aa61-6f0e-454e-9708-83413d640422\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") "
Sep 18 18:33:49.415203 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.415206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-webhook-cert\") pod \"f7c0aa61-6f0e-454e-9708-83413d640422\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") "
Sep 18 18:33:49.415397 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.415342    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-metrics-certs\") pod \"f7c0aa61-6f0e-454e-9708-83413d640422\" (UID: \"f7c0aa61-6f0e-454e-9708-83413d640422\") "
Sep 18 18:33:49.417339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.417307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7c0aa61-6f0e-454e-9708-83413d640422" (UID: "f7c0aa61-6f0e-454e-9708-83413d640422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:33:49.417491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.417392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c0aa61-6f0e-454e-9708-83413d640422-kube-api-access-jpfmg" (OuterVolumeSpecName: "kube-api-access-jpfmg") pod "f7c0aa61-6f0e-454e-9708-83413d640422" (UID: "f7c0aa61-6f0e-454e-9708-83413d640422"). InnerVolumeSpecName "kube-api-access-jpfmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:33:49.417491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.417448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f7c0aa61-6f0e-454e-9708-83413d640422" (UID: "f7c0aa61-6f0e-454e-9708-83413d640422"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:33:49.516753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.516697    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpfmg\" (UniqueName: \"kubernetes.io/projected/f7c0aa61-6f0e-454e-9708-83413d640422-kube-api-access-jpfmg\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:33:49.516753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.516717    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-webhook-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:33:49.516753 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.516726    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c0aa61-6f0e-454e-9708-83413d640422-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:33:49.735445 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.735408    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7c0aa61-6f0e-454e-9708-83413d640422" containerID="4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b" exitCode=0
Sep 18 18:33:49.735603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.735455    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8"
Sep 18 18:33:49.735603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.735509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8" event={"ID":"f7c0aa61-6f0e-454e-9708-83413d640422","Type":"ContainerDied","Data":"4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b"}
Sep 18 18:33:49.735603 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.735575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-5v5x8" event={"ID":"f7c0aa61-6f0e-454e-9708-83413d640422","Type":"ContainerDied","Data":"7ff8b6e720e6d5b0eebafb0a46583a1854c8efac2cfbcc533429673de3811c18"}
Sep 18 18:33:49.735792 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.735604    2580 scope.go:117] "RemoveContainer" containerID="4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b"
Sep 18 18:33:49.745341 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.745325    2580 scope.go:117] "RemoveContainer" containerID="4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b"
Sep 18 18:33:49.745606 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:33:49.745591    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b\": container with ID starting with 4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b not found: ID does not exist" containerID="4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b"
Sep 18 18:33:49.745669 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.745615    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b"} err="failed to get container status \"4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b\": rpc error: code = NotFound desc = could not find container \"4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b\": container with ID starting with 4924840f962103fd74ff79eae07a590007992b6cff693770bc36143be20d309b not found: ID does not exist"
Sep 18 18:33:49.755724 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.755704    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-5v5x8"]
Sep 18 18:33:49.761049 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.760989    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-5v5x8"]
Sep 18 18:33:49.910291 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:49.910262    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c0aa61-6f0e-454e-9708-83413d640422" path="/var/lib/kubelet/pods/f7c0aa61-6f0e-454e-9708-83413d640422/volumes"
Sep 18 18:33:50.418828 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.418786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:33:50.526514 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526482    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-vnkf6"]
Sep 18 18:33:50.526823 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526812    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c0aa61-6f0e-454e-9708-83413d640422" containerName="manager"
Sep 18 18:33:50.526868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526825    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c0aa61-6f0e-454e-9708-83413d640422" containerName="manager"
Sep 18 18:33:50.526868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526835    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy"
Sep 18 18:33:50.526868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526841    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab1e956-1d53-46ec-8b80-d40a22cff42d" containerName="istio-proxy"
Sep 18 18:33:50.526868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy"
Sep 18 18:33:50.526868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy"
Sep 18 18:33:50.527038 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526910    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee0eb77-299e-4050-b23b-7ef359bc26bb" containerName="istio-proxy"
Sep 18 18:33:50.527038 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.526918    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c0aa61-6f0e-454e-9708-83413d640422" containerName="manager"
Sep 18 18:33:50.531273 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.531254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.534434 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.534413    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 18:33:50.534557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.534435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xxjjv\""
Sep 18 18:33:50.534557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.534435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 18:33:50.534557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.534413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 18:33:50.540882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.540860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-vnkf6"]
Sep 18 18:33:50.625597 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.625570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4345be0-0731-4efb-b7f5-17b4848e4df9-metrics-certs\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.625720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.625621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4345be0-0731-4efb-b7f5-17b4848e4df9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.625720 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.625680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjdh\" (UniqueName: \"kubernetes.io/projected/a4345be0-0731-4efb-b7f5-17b4848e4df9-kube-api-access-bbjdh\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.726861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.726791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4345be0-0731-4efb-b7f5-17b4848e4df9-metrics-certs\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.726861 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.726842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4345be0-0731-4efb-b7f5-17b4848e4df9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.727064 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.726878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjdh\" (UniqueName: \"kubernetes.io/projected/a4345be0-0731-4efb-b7f5-17b4848e4df9-kube-api-access-bbjdh\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.729465 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.729439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4345be0-0731-4efb-b7f5-17b4848e4df9-webhook-cert\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.729581 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.729561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4345be0-0731-4efb-b7f5-17b4848e4df9-metrics-certs\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.734905 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.734880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjdh\" (UniqueName: \"kubernetes.io/projected/a4345be0-0731-4efb-b7f5-17b4848e4df9-kube-api-access-bbjdh\") pod \"maas-controller-7dc7cc4dff-vnkf6\" (UID: \"a4345be0-0731-4efb-b7f5-17b4848e4df9\") " pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:50.843181 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:50.843154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:51.112208 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:51.112183    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dc7cc4dff-vnkf6"]
Sep 18 18:33:51.114436 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:33:51.114405    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4345be0_0731_4efb_b7f5_17b4848e4df9.slice/crio-ad10cd33b7a9ea4015ce1a6a68ee3e50e2033b44931aadd8cd109f0b8cb3a52f WatchSource:0}: Error finding container ad10cd33b7a9ea4015ce1a6a68ee3e50e2033b44931aadd8cd109f0b8cb3a52f: Status 404 returned error can't find the container with id ad10cd33b7a9ea4015ce1a6a68ee3e50e2033b44931aadd8cd109f0b8cb3a52f
Sep 18 18:33:51.752664 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:51.752583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6" event={"ID":"a4345be0-0731-4efb-b7f5-17b4848e4df9","Type":"ContainerStarted","Data":"aaddd7df09211f4ead63b168c01f6d1956e84ad6e7226e1d210a0f135a520353"}
Sep 18 18:33:51.752664 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:51.752637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6" event={"ID":"a4345be0-0731-4efb-b7f5-17b4848e4df9","Type":"ContainerStarted","Data":"ad10cd33b7a9ea4015ce1a6a68ee3e50e2033b44931aadd8cd109f0b8cb3a52f"}
Sep 18 18:33:51.753125 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:51.752703    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:33:51.770653 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:33:51.770606    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6" podStartSLOduration=1.398114754 podStartE2EDuration="1.770591542s" totalImagesPullingTime="372.476788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:33:51.76883483 +0000 UTC m=+2001.920709666" watchObservedRunningTime="2026-09-18 18:33:51.770591542 +0000 UTC m=+2001.922466377"
Sep 18 18:34:02.761494 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:02.761460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dc7cc4dff-vnkf6"
Sep 18 18:34:54.805703 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:54.805565    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"]
Sep 18 18:34:54.806316 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:54.805799    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h" podUID="1426b077-07c1-43a9-a1df-9bac8fd86d4f" containerName="manager" containerID="cri-o://837435882b7880be327e0488c39addb97ebffacfc32d5489163e4ffc5ebab4d0" gracePeriod=10
Sep 18 18:34:56.054948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.054865    2580 generic.go:356] "Generic (PLEG): container finished" podID="1426b077-07c1-43a9-a1df-9bac8fd86d4f" containerID="837435882b7880be327e0488c39addb97ebffacfc32d5489163e4ffc5ebab4d0" exitCode=0
Sep 18 18:34:56.054948 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.054924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h" event={"ID":"1426b077-07c1-43a9-a1df-9bac8fd86d4f","Type":"ContainerDied","Data":"837435882b7880be327e0488c39addb97ebffacfc32d5489163e4ffc5ebab4d0"}
Sep 18 18:34:56.346383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.346357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:34:56.468236 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.468207    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f85jc\" (UniqueName: \"kubernetes.io/projected/1426b077-07c1-43a9-a1df-9bac8fd86d4f-kube-api-access-f85jc\") pod \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") "
Sep 18 18:34:56.468395 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.468270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1426b077-07c1-43a9-a1df-9bac8fd86d4f-extensions-socket-volume\") pod \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\" (UID: \"1426b077-07c1-43a9-a1df-9bac8fd86d4f\") "
Sep 18 18:34:56.468605 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.468583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1426b077-07c1-43a9-a1df-9bac8fd86d4f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1426b077-07c1-43a9-a1df-9bac8fd86d4f" (UID: "1426b077-07c1-43a9-a1df-9bac8fd86d4f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:34:56.470415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.470396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1426b077-07c1-43a9-a1df-9bac8fd86d4f-kube-api-access-f85jc" (OuterVolumeSpecName: "kube-api-access-f85jc") pod "1426b077-07c1-43a9-a1df-9bac8fd86d4f" (UID: "1426b077-07c1-43a9-a1df-9bac8fd86d4f"). InnerVolumeSpecName "kube-api-access-f85jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:34:56.569426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.569401    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1426b077-07c1-43a9-a1df-9bac8fd86d4f-extensions-socket-volume\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:34:56.569426 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:56.569422    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85jc\" (UniqueName: \"kubernetes.io/projected/1426b077-07c1-43a9-a1df-9bac8fd86d4f-kube-api-access-f85jc\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:34:57.060495 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:57.060467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"
Sep 18 18:34:57.060901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:57.060509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h" event={"ID":"1426b077-07c1-43a9-a1df-9bac8fd86d4f","Type":"ContainerDied","Data":"f47aeefb24b95659cc100d5c0d7eac713fed2ea8407519ac1cfd5a86132794d6"}
Sep 18 18:34:57.060901 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:57.060568    2580 scope.go:117] "RemoveContainer" containerID="837435882b7880be327e0488c39addb97ebffacfc32d5489163e4ffc5ebab4d0"
Sep 18 18:34:57.081533 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:57.081512    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"]
Sep 18 18:34:57.086969 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:57.086947    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4s45h"]
Sep 18 18:34:57.910111 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:34:57.910074    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1426b077-07c1-43a9-a1df-9bac8fd86d4f" path="/var/lib/kubelet/pods/1426b077-07c1-43a9-a1df-9bac8fd86d4f/volumes"
Sep 18 18:35:27.119459 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:27.119415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:35:55.516732 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.516702    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"]
Sep 18 18:35:55.517200 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.517067    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1426b077-07c1-43a9-a1df-9bac8fd86d4f" containerName="manager"
Sep 18 18:35:55.517200 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.517080    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1426b077-07c1-43a9-a1df-9bac8fd86d4f" containerName="manager"
Sep 18 18:35:55.517200 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.517131    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1426b077-07c1-43a9-a1df-9bac8fd86d4f" containerName="manager"
Sep 18 18:35:55.519962 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.519943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.522076 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.522058    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j6zlm\""
Sep 18 18:35:55.533156 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.533130    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"]
Sep 18 18:35:55.645091 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.645068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmbn\" (UniqueName: \"kubernetes.io/projected/8d04de07-9fcc-463b-bccb-2584838a7f3e-kube-api-access-stmbn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mvqlk\" (UID: \"8d04de07-9fcc-463b-bccb-2584838a7f3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.645211 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.645133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d04de07-9fcc-463b-bccb-2584838a7f3e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mvqlk\" (UID: \"8d04de07-9fcc-463b-bccb-2584838a7f3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.745749 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.745718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d04de07-9fcc-463b-bccb-2584838a7f3e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mvqlk\" (UID: \"8d04de07-9fcc-463b-bccb-2584838a7f3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.745874 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.745777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmbn\" (UniqueName: \"kubernetes.io/projected/8d04de07-9fcc-463b-bccb-2584838a7f3e-kube-api-access-stmbn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mvqlk\" (UID: \"8d04de07-9fcc-463b-bccb-2584838a7f3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.746179 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.746156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d04de07-9fcc-463b-bccb-2584838a7f3e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mvqlk\" (UID: \"8d04de07-9fcc-463b-bccb-2584838a7f3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.753882 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.753863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmbn\" (UniqueName: \"kubernetes.io/projected/8d04de07-9fcc-463b-bccb-2584838a7f3e-kube-api-access-stmbn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mvqlk\" (UID: \"8d04de07-9fcc-463b-bccb-2584838a7f3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:55.833637 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:55.833551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:56.126900 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:56.126874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"]
Sep 18 18:35:56.129018 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:35:56.128987    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d04de07_9fcc_463b_bccb_2584838a7f3e.slice/crio-e126d31da35d6e2ca93821550345298baa05388cde0e619336af47d26929b096 WatchSource:0}: Error finding container e126d31da35d6e2ca93821550345298baa05388cde0e619336af47d26929b096: Status 404 returned error can't find the container with id e126d31da35d6e2ca93821550345298baa05388cde0e619336af47d26929b096
Sep 18 18:35:56.327544 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:56.327511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk" event={"ID":"8d04de07-9fcc-463b-bccb-2584838a7f3e","Type":"ContainerStarted","Data":"e4b0166da809b09c762de749ef9dfad1dca6f681933683fa89556746343e468c"}
Sep 18 18:35:56.327721 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:56.327562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk" event={"ID":"8d04de07-9fcc-463b-bccb-2584838a7f3e","Type":"ContainerStarted","Data":"e126d31da35d6e2ca93821550345298baa05388cde0e619336af47d26929b096"}
Sep 18 18:35:56.327721 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:56.327641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:35:56.348533 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:35:56.348152    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk" podStartSLOduration=1.348131563 podStartE2EDuration="1.348131563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:35:56.346973401 +0000 UTC m=+2126.498848237" watchObservedRunningTime="2026-09-18 18:35:56.348131563 +0000 UTC m=+2126.500006398"
Sep 18 18:36:07.333531 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:36:07.333497    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mvqlk"
Sep 18 18:36:21.283539 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:36:21.283437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:36:22.592477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:36:22.592439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:36:32.297036 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:36:32.297003    2580 scope.go:117] "RemoveContainer" containerID="cf7bfdfe5e6032fcaf1b8bb30d124f521472eacc19cbcbd3fb5d4a4f8d992cf1"
Sep 18 18:36:44.235055 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:36:44.229480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:36:48.866555 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:36:48.866519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:37:04.217292 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:04.217201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:37:18.469087 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:18.469052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:37:19.505879 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:19.505838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:37:33.810077 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.810017    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"]
Sep 18 18:37:33.813491 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.813471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.827831 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.823410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"]
Sep 18 18:37:33.868422 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.868387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7jw\" (UniqueName: \"kubernetes.io/projected/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-kube-api-access-pg7jw\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.868572 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.868448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-metrics-certs\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.868572 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.868467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-openshift-service-ca\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.868572 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.868484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-maas-api-tls\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.969383 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.969349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-metrics-certs\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.969570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.969388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-openshift-service-ca\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.969570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.969419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-maas-api-tls\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.969570 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.969463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7jw\" (UniqueName: \"kubernetes.io/projected/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-kube-api-access-pg7jw\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.970014 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.969992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-openshift-service-ca\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.972101 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.972071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-metrics-certs\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.972194 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.972139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-maas-api-tls\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:33.976974 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:33.976955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7jw\" (UniqueName: \"kubernetes.io/projected/1cb7eb7e-f433-4181-bf50-7dbe8405e3f7-kube-api-access-pg7jw\") pod \"maas-api-86d9ddc875-cglhm\" (UID: \"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:34.124590 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:34.124498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:34.397327 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:34.397242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"]
Sep 18 18:37:34.413393 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:37:34.413364    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb7eb7e_f433_4181_bf50_7dbe8405e3f7.slice/crio-ce57318f9e5e1d080071918693879cd01e0562a7b292f1f1bb11e05564b3bba4 WatchSource:0}: Error finding container ce57318f9e5e1d080071918693879cd01e0562a7b292f1f1bb11e05564b3bba4: Status 404 returned error can't find the container with id ce57318f9e5e1d080071918693879cd01e0562a7b292f1f1bb11e05564b3bba4
Sep 18 18:37:34.414724 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:34.414705    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:37:34.787257 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:34.787231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm" event={"ID":"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7","Type":"ContainerStarted","Data":"ce57318f9e5e1d080071918693879cd01e0562a7b292f1f1bb11e05564b3bba4"}
Sep 18 18:37:35.793080 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:35.793050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm" event={"ID":"1cb7eb7e-f433-4181-bf50-7dbe8405e3f7","Type":"ContainerStarted","Data":"16381957fc2601766a5db7af3e132cff5bc2efde2249e25f4f091df20c3f6843"}
Sep 18 18:37:35.793456 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:35.793148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:35.811435 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:35.811388    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm" podStartSLOduration=2.500708135 podStartE2EDuration="2.811371786s" totalImagesPullingTime="310.663651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:37:35.80989476 +0000 UTC m=+2225.961769596" watchObservedRunningTime="2026-09-18 18:37:35.811371786 +0000 UTC m=+2225.963246621"
Sep 18 18:37:41.802839 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:41.802810    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cglhm"
Sep 18 18:37:41.850393 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:41.850363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"]
Sep 18 18:37:41.850633 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:41.850608    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr" podUID="e873313f-0af1-4eec-9b30-e8019da9da23" containerName="maas-api" containerID="cri-o://61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f" gracePeriod=30
Sep 18 18:37:43.086443 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.086409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:37:43.155386 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.155353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-maas-api-tls\") pod \"e873313f-0af1-4eec-9b30-e8019da9da23\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") "
Sep 18 18:37:43.155540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.155393    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rgl2\" (UniqueName: \"kubernetes.io/projected/e873313f-0af1-4eec-9b30-e8019da9da23-kube-api-access-7rgl2\") pod \"e873313f-0af1-4eec-9b30-e8019da9da23\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") "
Sep 18 18:37:43.155540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.155423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873313f-0af1-4eec-9b30-e8019da9da23-openshift-service-ca\") pod \"e873313f-0af1-4eec-9b30-e8019da9da23\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") "
Sep 18 18:37:43.155540 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.155446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-metrics-certs\") pod \"e873313f-0af1-4eec-9b30-e8019da9da23\" (UID: \"e873313f-0af1-4eec-9b30-e8019da9da23\") "
Sep 18 18:37:43.155816 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.155784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e873313f-0af1-4eec-9b30-e8019da9da23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e873313f-0af1-4eec-9b30-e8019da9da23" (UID: "e873313f-0af1-4eec-9b30-e8019da9da23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:37:43.157696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.157672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e873313f-0af1-4eec-9b30-e8019da9da23" (UID: "e873313f-0af1-4eec-9b30-e8019da9da23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:37:43.157801 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.157706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e873313f-0af1-4eec-9b30-e8019da9da23" (UID: "e873313f-0af1-4eec-9b30-e8019da9da23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:37:43.157883 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.157854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e873313f-0af1-4eec-9b30-e8019da9da23-kube-api-access-7rgl2" (OuterVolumeSpecName: "kube-api-access-7rgl2") pod "e873313f-0af1-4eec-9b30-e8019da9da23" (UID: "e873313f-0af1-4eec-9b30-e8019da9da23"). InnerVolumeSpecName "kube-api-access-7rgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:37:43.256582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.256522    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-maas-api-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:37:43.256582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.256548    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rgl2\" (UniqueName: \"kubernetes.io/projected/e873313f-0af1-4eec-9b30-e8019da9da23-kube-api-access-7rgl2\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:37:43.256582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.256557    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873313f-0af1-4eec-9b30-e8019da9da23-openshift-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:37:43.256582 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.256566    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873313f-0af1-4eec-9b30-e8019da9da23-metrics-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Sep 18 18:37:43.838495 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.838459    2580 generic.go:356] "Generic (PLEG): container finished" podID="e873313f-0af1-4eec-9b30-e8019da9da23" containerID="61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f" exitCode=0
Sep 18 18:37:43.838690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.838518    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"
Sep 18 18:37:43.838690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.838541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr" event={"ID":"e873313f-0af1-4eec-9b30-e8019da9da23","Type":"ContainerDied","Data":"61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f"}
Sep 18 18:37:43.838690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.838626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr" event={"ID":"e873313f-0af1-4eec-9b30-e8019da9da23","Type":"ContainerDied","Data":"482a4d981be5f6098358d2b73b9d8577b6955956fb525b47a3ec32bc1ef60fc5"}
Sep 18 18:37:43.838690 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.838655    2580 scope.go:117] "RemoveContainer" containerID="61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f"
Sep 18 18:37:43.848931 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.848912    2580 scope.go:117] "RemoveContainer" containerID="61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f"
Sep 18 18:37:43.849229 ip-10-0-136-64 kubenswrapper[2580]: E0918 18:37:43.849210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f\": container with ID starting with 61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f not found: ID does not exist" containerID="61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f"
Sep 18 18:37:43.849297 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.849236    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f"} err="failed to get container status \"61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f\": rpc error: code = NotFound desc = could not find container \"61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f\": container with ID starting with 61474a2ba62ff897bed97dbf6de57bf4e82579c4cb0ff3524cb54a8617be725f not found: ID does not exist"
Sep 18 18:37:43.859539 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.859519    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"]
Sep 18 18:37:43.863730 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.863707    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vndcr"]
Sep 18 18:37:43.909186 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:43.909165    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e873313f-0af1-4eec-9b30-e8019da9da23" path="/var/lib/kubelet/pods/e873313f-0af1-4eec-9b30-e8019da9da23/volumes"
Sep 18 18:37:45.465868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:37:45.465834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:38:04.266785 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:04.266748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:38:04.368265 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:04.368235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:38:07.347655 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:07.347606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qh4kd_d0daa103-d922-4223-bf67-567a3fb989c5/global-pull-secret-syncer/0.log"
Sep 18 18:38:07.401143 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:07.401115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log"
Sep 18 18:38:07.533134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:07.533104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-64.ec2.internal_4547ca9ec435802628cd2ede92cc0a62/haproxy/0.log"
Sep 18 18:38:12.304888 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:12.304808    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-mvqlk_8d04de07-9fcc-463b-bccb-2584838a7f3e/manager/0.log"
Sep 18 18:38:12.405873 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:12.405819    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6qj4f_cf1d2fb1-f18d-4d2b-9fc8-5ec8b3474022/manager/0.log"
Sep 18 18:38:13.987568 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:13.987516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78ccbd98dc-vnnh6_5b5385dc-804e-4017-b8c5-46e93f4c5d2f/metrics-server/0.log"
Sep 18 18:38:14.038964 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.038934    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7q9vs_0eca1198-c433-41fd-a99a-fcdb01a75b2f/node-exporter/0.log"
Sep 18 18:38:14.056083 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.056053    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7q9vs_0eca1198-c433-41fd-a99a-fcdb01a75b2f/kube-rbac-proxy/0.log"
Sep 18 18:38:14.075862 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.075847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7q9vs_0eca1198-c433-41fd-a99a-fcdb01a75b2f/init-textfile/0.log"
Sep 18 18:38:14.356445 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.356420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/prometheus/0.log"
Sep 18 18:38:14.377838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.377800    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/config-reloader/0.log"
Sep 18 18:38:14.415415 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.415391    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/thanos-sidecar/0.log"
Sep 18 18:38:14.450868 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.450800    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/kube-rbac-proxy-web/0.log"
Sep 18 18:38:14.470124 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.470105    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/kube-rbac-proxy/0.log"
Sep 18 18:38:14.501160 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.501139    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/kube-rbac-proxy-thanos/0.log"
Sep 18 18:38:14.523637 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.523613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda83ab2-1ada-402e-9ee8-d8aa4a72a181/init-config-reloader/0.log"
Sep 18 18:38:14.752520 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.752450    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f948dfbf4-h9769_300822b8-6ebc-41d7-a2e3-e9e96030d4d9/thanos-query/0.log"
Sep 18 18:38:14.773503 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.773479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f948dfbf4-h9769_300822b8-6ebc-41d7-a2e3-e9e96030d4d9/kube-rbac-proxy-web/0.log"
Sep 18 18:38:14.793342 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.793316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f948dfbf4-h9769_300822b8-6ebc-41d7-a2e3-e9e96030d4d9/kube-rbac-proxy/0.log"
Sep 18 18:38:14.811579 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.811558    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f948dfbf4-h9769_300822b8-6ebc-41d7-a2e3-e9e96030d4d9/prom-label-proxy/0.log"
Sep 18 18:38:14.830131 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.830113    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f948dfbf4-h9769_300822b8-6ebc-41d7-a2e3-e9e96030d4d9/kube-rbac-proxy-rules/0.log"
Sep 18 18:38:14.848238 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:14.848214    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f948dfbf4-h9769_300822b8-6ebc-41d7-a2e3-e9e96030d4d9/kube-rbac-proxy-metrics/0.log"
Sep 18 18:38:17.359384 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.359345    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"]
Sep 18 18:38:17.359855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.359674    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e873313f-0af1-4eec-9b30-e8019da9da23" containerName="maas-api"
Sep 18 18:38:17.359855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.359685    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e873313f-0af1-4eec-9b30-e8019da9da23" containerName="maas-api"
Sep 18 18:38:17.359855 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.359744    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e873313f-0af1-4eec-9b30-e8019da9da23" containerName="maas-api"
Sep 18 18:38:17.362671 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.362648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.364598 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.364578    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x8m5v\"/\"kube-root-ca.crt\""
Sep 18 18:38:17.364696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.364615    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x8m5v\"/\"default-dockercfg-qmjsc\""
Sep 18 18:38:17.364696 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.364687    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x8m5v\"/\"openshift-service-ca.crt\""
Sep 18 18:38:17.371533 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.371511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"]
Sep 18 18:38:17.443740 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.443708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-lib-modules\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.443889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.443758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-sys\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.443889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.443781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-proc\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.443889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.443821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqp46\" (UniqueName: \"kubernetes.io/projected/47e8b60d-806b-41d8-85b6-69416bb40f3b-kube-api-access-rqp46\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.443889 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.443842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-podres\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545137 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-proc\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqp46\" (UniqueName: \"kubernetes.io/projected/47e8b60d-806b-41d8-85b6-69416bb40f3b-kube-api-access-rqp46\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-podres\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-proc\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-lib-modules\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545339 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-sys\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-podres\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-sys\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.545563 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.545420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47e8b60d-806b-41d8-85b6-69416bb40f3b-lib-modules\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.552828 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.552798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqp46\" (UniqueName: \"kubernetes.io/projected/47e8b60d-806b-41d8-85b6-69416bb40f3b-kube-api-access-rqp46\") pod \"perf-node-gather-daemonset-vtnlc\" (UID: \"47e8b60d-806b-41d8-85b6-69416bb40f3b\") " pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.673477 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.673450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:17.946450 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.946422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"]
Sep 18 18:38:17.949235 ip-10-0-136-64 kubenswrapper[2580]: W0918 18:38:17.949213    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod47e8b60d_806b_41d8_85b6_69416bb40f3b.slice/crio-aced02cd77f3738d03e6b8e13ada6c85707ef2fa66d5bd0df531a1a551a08f16 WatchSource:0}: Error finding container aced02cd77f3738d03e6b8e13ada6c85707ef2fa66d5bd0df531a1a551a08f16: Status 404 returned error can't find the container with id aced02cd77f3738d03e6b8e13ada6c85707ef2fa66d5bd0df531a1a551a08f16
Sep 18 18:38:17.990842 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:17.990819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc" event={"ID":"47e8b60d-806b-41d8-85b6-69416bb40f3b","Type":"ContainerStarted","Data":"aced02cd77f3738d03e6b8e13ada6c85707ef2fa66d5bd0df531a1a551a08f16"}
Sep 18 18:38:18.197646 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:18.197572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mchr8_62d5c61a-8176-416d-874b-b61ae7f28b64/konnectivity-agent/0.log" limit=1024
Sep 18 18:38:18.996921 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:18.996879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc" event={"ID":"47e8b60d-806b-41d8-85b6-69416bb40f3b","Type":"ContainerStarted","Data":"4d0f63b35330ae1b0642cfa50a4a60199c24a5c1a80d7c86c29424baed38a598"}
Sep 18 18:38:18.997368 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:18.996976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:19.014642 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:19.014585    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc" podStartSLOduration=2.014570297 podStartE2EDuration="2.014570297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:38:19.013730822 +0000 UTC m=+2269.165605658" watchObservedRunningTime="2026-09-18 18:38:19.014570297 +0000 UTC m=+2269.166445132"
Sep 18 18:38:19.658599 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:19.658567    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnwcp_542cb77a-84d0-433d-9d8f-36cb17a660f2/dns/0.log"
Sep 18 18:38:19.677347 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:19.677319    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnwcp_542cb77a-84d0-433d-9d8f-36cb17a660f2/kube-rbac-proxy/0.log"
Sep 18 18:38:19.764695 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:19.764671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jjslp_9e980d25-4957-4cfd-899d-8c31cd4786dd/dns-node-resolver/0.log"
Sep 18 18:38:20.349134 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:20.349105    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxk9w_d685ebef-6279-418c-ba6e-dbb5e8b24bd0/node-ca/0.log"
Sep 18 18:38:22.643454 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:22.643422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmzmg_b65eb7ba-3c83-4ff3-acee-793412b227c5/serve-healthcheck-canary/0.log"
Sep 18 18:38:23.252660 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:23.252631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtwbq_a57d4563-a4fe-4512-92de-b33ab932fc1b/kube-rbac-proxy/0.log"
Sep 18 18:38:23.272989 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:23.272961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtwbq_a57d4563-a4fe-4512-92de-b33ab932fc1b/exporter/0.log"
Sep 18 18:38:23.292804 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:23.292772    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtwbq_a57d4563-a4fe-4512-92de-b33ab932fc1b/extractor/0.log"
Sep 18 18:38:25.011557 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.011531    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x8m5v/perf-node-gather-daemonset-vtnlc"
Sep 18 18:38:25.331838 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.331756    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7747fb5fbd-28sv2_8c063afa-b3c7-4d78-8dd9-dd874ee7797f/manager/0.log"
Sep 18 18:38:25.353709 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.353682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cmr2b_f8842d59-6c7a-446a-a0ea-7cfa9702ec4e/manager/0.log"
Sep 18 18:38:25.767843 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.767804    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pdpc5_d9d560de-0a4a-4a13-ae2d-12a76fa58452/manager/0.log"
Sep 18 18:38:25.840966 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.840933    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dc7cc4dff-vnkf6_a4345be0-0731-4efb-b7f5-17b4848e4df9/manager/0.log"
Sep 18 18:38:25.970070 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.970018    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57cbb766c6-45zzb_c2f252eb-615a-453d-b179-9a17d7c25f27/manager/0.log"
Sep 18 18:38:25.987194 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:25.987175    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57cbb766c6-45zzb_c2f252eb-615a-453d-b179-9a17d7c25f27/copy-manifests/0.log"
Sep 18 18:38:27.588448 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:27.588411    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d5f876c6c-62fp8_0c13bc09-19cd-4716-9519-653fdf1442ee/manager/0.log"
Sep 18 18:38:34.050460 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.050429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/kube-multus-additional-cni-plugins/0.log"
Sep 18 18:38:34.069249 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.069224    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/egress-router-binary-copy/0.log"
Sep 18 18:38:34.087588 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.087566    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/cni-plugins/0.log"
Sep 18 18:38:34.106055 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.106009    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/bond-cni-plugin/0.log"
Sep 18 18:38:34.125370 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.125348    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/routeoverride-cni/0.log"
Sep 18 18:38:34.143224 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.143208    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/whereabouts-cni-bincopy/0.log"
Sep 18 18:38:34.163405 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.163381    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74hfr_5c8bc962-f106-4454-bbfc-3bffe3b209b6/whereabouts-cni/0.log"
Sep 18 18:38:34.406960 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.406880    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s8h5t_4ebd8a70-1851-4dc9-89dd-f56715c00883/kube-multus/0.log"
Sep 18 18:38:34.480975 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.480951    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bs8m5_4ce9057c-cad7-4604-9e93-b63faa1362ba/network-metrics-daemon/0.log"
Sep 18 18:38:34.498523 ip-10-0-136-64 kubenswrapper[2580]: I0918 18:38:34.498503    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bs8m5_4ce9057c-cad7-4604-9e93-b63faa1362ba/kube-rbac-proxy/0.log"